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ABSTRACT 



This report describes the results from an exploratory 
project conducted for the National Center for Education Statistics. The 
purpose of the project was to develop and field test questionnaire items and 
related methods designed to capture information about the instructional 
processes used nationally in 8th- to 12th-grade mathematics classrooms. Data 
from classroom observers, short logs of daily classroom practice, and a 
teacher questionnaire were used to assess the quality of the items. A survey 
questionnaire, field tested with 100 teachers, was completed by approximately 
300 mathematics teachers, and a case study was conducted with 41 teachers in 
similar settings with comparable responsibilities. Results of these studies 
were used to make 10 recommendations to improve the field test validation 
process and the survey items. Thirteen appendixes contain supplemental 
information about the study and the test items. Appendix M contains a 
summary, in chart form, of recommended changes to the field test items. 
(Contains 21 references.) (SLD) 
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This report describes the results from an exploratory project conducted for the 
National Center for Education Statistics (NCES). The purpose of the project was to develop 
and fieldtest questionnaire items and related methods designed to capture information 
about the instructional processes used nationally in eighth to twelfth grade mathematics 
classrooms. Data from classroom observers, short logs of daily classroom practice kept over 
a month, and a teacher questionnaire were used to assess the quality of the items. Such 
instruments and methods can inform our understanding of options for collecting and 
validating survey data that could be incorporated into national data collection schemes. 
Knowing the reliability of self-report items can create confidence that the picture of 
instruction emerging from teacher surveys is likely to be accurate. 

The project had two parts: (1) a survey questionnaire completed by approximately 
300 mathematics teachers of eighth to twelfth grade students in randomly selected public 
and private schools and (2) a case study of approximately 40 teachers in similar settings 
with comparable responsibilities. The questionnaire, fieldtested with 100 teachers in an 
earlier pilot project, built on previous studies including the Third International 
Mathematics and Science Study (TIMSS), the UCLA/RAND Validating National Curriculum 
Indicators project, and Reform Up Close. The case study teachers chose one designated 
class on their schedule. For this class, they completed pre- and post-study questionnaires, 
were observed by researchers, participated in a follow-up interview, and completed four 
weeks of daily log forms detailing the instructional practices they used. Surveys and case 
studies provide information about the extent to which the proposed items generate data that 
accurately portray instructional experiences in eighth to twelfth grade mathematics classes. 

This effort is part of a series of activities through which NCES is examining and 
refining data collection methods and instruments suitable for use in national programs. 
Surveys are among the most cost-effective and least burdensome methods, but they may not 
produce an accurate and reliable picture of instruction. This project was designed to 
improve our ability to gather more useful information about practice without adding to 
respondents’ burden in providing data. 



Background 



NCES collects and publishes a broad array of policy-relevant information on the 
condition of education: school organization and facilities; the professional characteristics of 
principals, teachers, and other school staff; and aggregate demand for teachers by areas of 
specialization. Current and recent NCES data collections on K-12 education (or selected 



grade ranges) include the Schools and Staffing Survey (SASS), the Fast Response Survey 
System, the National Assessment of Educational Progress, and the National Education 
Longitudinal Study of 1988. 

However, missing from this array is similarly comprehensive and nationally- 
representative data that describe the real work of schools, teachers, and students. What 
goes on inside classrooms when teachers and students buckle down, close the door, and get 
to work? What types of teaching occur in schools? What is the relative instructional 
emphasis on broad concepts compared with specific facts? To what extent do teachers use 
newly-recommended instructional techniques rather than the “tried and true” methods with 
long traditions of use? How are students assessed? Who determines policy on homework, 
student discipline, and promotions? 

This information is important because issues like these and others explicitly affect 
the quality of students’ experiences in the classroom and school and presumably bear 
directly on student learning. To date, most detailed pictures of classroom processes and 
participants come from case study data collected in a small number of educational settings. 
That information, while often qualitatively rich, provides only a non-representative glimpse 
into a few classrooms — often fascinating and sometimes riveting, but always limited in 
scope. We can make no assumptions about the extent to which those pictures are repeated 
in education settings that vary widely throughout the country. 

Continuing emphasis on educational accountability and productivity has heightened 
interest in measuring the distinct contribution of schooling to achievement. As high-stakes 
testing becomes more prominent, the desire to understand variation in achievement among 
students and between groups of students (by race or ethnicity, socio-economic status, or 
geographic region) will focus attention on how classroom instructional practices and the 
resulting opportunities for students to learn the material being assessed vary within and 
among schools. 

Stodolsky (1996), summarized the rationale for generating a more broadly 
representative, yet finely textured data base: 

If we are to understand, monitor, and improve our nation’s schools, accurate 
and timely empirical, descriptive data about how schools work must be 
available. The activities that take place in classrooms to engender student 
learning and development are the heart of any school’s education efforts. It is 
in the transactions between and among teachers, students, materials, and 
tasks that deliberate efforts to educate occur. Descriptive information about 



how teaching and learning occur in classrooms and about what is taught 
provides the basis for monitoring the status of instruction in a large number 
of settings. Such information can provide periodic assessments of stability 
and change in instruction, particularly as changes relate to deliberate efforts 
to reform or alter curriculum and instruction, (p. 1-2) 

Student achievement reflects— among other factors— the opportunities to learn 
provided by classroom experiences. Those classroom experiences include the content 
covered; the learning objectives enacted; and the instruction offered to achieve those 
objectives. The intended curriculum describes the outer limits of what we might reasonably 
expect students to learn; the enacted curriculum encompasses the content and instructional 
practices actually experienced by students; and the attained curriculum is that which the 
student can demonstrate knowing. 

It is not sufficient to know only the level at which students achieve, since that also 
reflects students’ prior learning and social conditioning (Porter, 1993; Berk, 1988). Nor does 
knowing only the gain in student learning over a particular period of time provide the 
information necessary to understand the events that precipitated learning. However, 
understanding how variation in student learning relates to variation in instructional 
practices could inform local, state, and national educational policy (Burstein, Oakes, Guiton, 
1992; Smith 1988; Murnane, 1987). Because survey data are likely to be the major source of 
nationally-representative information about instructional content and practices, the quality 
of survey items should be initially validated and periodically confirmed (Burstein, 
McDonnell, Winkle, Ormseth, Mirocha, Guiton, 1995). 

NCES began this series of studies of how to collect such information in 1994 with a 
comprehensive review of current research efforts and instruments (Leighton, Mullens, 
Turnbull, Weiner, & Williams, 1995). This provided the foundation for a 1995 project to 
review existing measurement approaches (Mullens, 1995) and to develop a module of items 
measuring classroom instructional processes for the Current Teachers Questionnaire of the 
SASS Teacher Follow-up Survey (TFS). That module included only questions that were 
applicable to the broad range of grades, circumstances, and content areas from which the 
sample of TFS respondents is drawn. This kept the focus broad and limited the extent to 
which more detailed information relevant to learning outcomes could be gathered. Related 
work revised those items to address data collection on instructional processes used by 
teachers of eighth to tenth graders in an effort to trade breadth for depth; narrowing the 
respondent range to one subject and a cluster of grades allowed inclusion of items eliciting 
greater detail about instruction. Those items were fieldtested with 100 teachers in three 
districts (Mullens and Leighton, 1996). Results from that project and subsequent 



experience guided our questionnaire revisions and planning for its fieldtest. The methods, 
instruments, items, and validation strategies we used are extensions of those developed 
(and in some cases under development still) by researchers working on TIMSS (1998, 1996), 
the UCLA/RAND Validating National Curriculum Indicators project (1995), and Reform Up 
Close (1993). 



Project Description 



This project was designed to field test revised items on classroom instructional 
practices. Two main questions governed data collection: 

1 . How accurate and reliable are the mail survey data that can be collected from 
secondary mathematics teachers about their instructional practices and the 
contexts within which they occur? 

2. Which of the revised items hold the most promise for large-scale use? 

To answer these questions, we developed a teacher questionnaire and then used case 
studies to assess the quality of the teacher questionnaire data. In developing the items, we 
used methods developed by Porter and his colleagues to identify the effects of increased 
enrollments on the content and pedagogy of high school mathematics and science courses 
(1993). We also drew heavily on the work of Burstein, McDonnell, et. al. (1995), which 
began developing validation procedures to improve the quality of national indicators of 
curriculum. 



Questionnaire Content 

The survey included 19 items, most with subitems, covering four areas of 
instructional practice: (1) conditions for teaching and learning in the school and classroom; 
(2) course content and emphasis; (3) instruction; and (4) the availability and use of 
instructional resources. Some items in the “conditions” section collected information about 
the length and frequency of the class meetings, the number and grade levels of the students 
enrolled, their ability levels compared to others in the school, and the teacher’s assessment 
of student capability to learn the course material, among other things. All are factors that 
affect teaching. The other items asked about the teacher’s role in setting school policies and 
control over classroom events. Such factors affect classroom learning because they affect 



what teachers choose to do for their students. Accordingly, researchers and others studying 
variation in classroom instructional practices frequently look for data on these factors to 
establish the context within which those decisions are made. 

Survey items about the course collected information about the specific topics covered, 
the level of emphasis teachers place on certain skills and concepts, student learning 
objectives, assessment content, integration with other courses or subjects, and nonacademic 
time. This information offers an explanatory context for teacher responses to questions on 
instruction and instructional resources. Since previous studies have shown that course 
titles provide little information about the specific academic topics covered, one item asked 
for the specific content covered. 

Items on instructional techniques formed the core of the questionnaire. One asked 
about the frequency with which teachers use various instructional methods in the target 
class. It included twelve activities, ranging from those commonly associated with 
traditional teaching (such as lecture and student recitation or drill) , to those associated with 
the National Council of Teachers of Mathematics (NCTM) recommendations (such as 
student discussions of approaches to solving problems, explanations of mathematical 
thinking and open-ended questions) , and those common to a range of styles (such as giving 
tests). A companion item focused on 24 student lesson activities, similarly distributed 
among traditional approaches (e.g., working on exercises, worksheets, or workbooks), NCTM 
recommendations (e.g., doing projects/assignments that take a week or more to finish), and 
cross-cutting activities (e.g., correcting or reviewing previous day’s homework). Finally, 
teachers were asked to identify the formal and informal ways in which they had students 
demonstrate proficiency. The options ranged from memorizing facts or formulae to 
generating original examples of mathematical concepts. 

Having appropriate materials and the skills to use them affects the choices teachers 
make about instruction. Therefore, two nearly identical items collected information from 
teachers about the availability of various materials and equipment during classroom 
instruction and their use of those items, and the extent to which the same materials were 
available to and used by students. 



The questionnaire is in Appendix A. 



Mail Survey 



We conducted the mail survey during a two-month period between April 16 and June 
23, 1997. Using lists of public and private secondary schools that were randomly drawn 
from three census categories (central city, urban fringe/large town, and rural/small town), 
we created a sample of 140 schools. 1 To the district superintendent of the selected public 
schools, we sent an explanatory letter, enclosing information about the fieldtest and 
advising them we would be contacting the school’s principal. We also sent explanatory 
letters to the principals of the schools and followed up with telephone calls to answer 
questions and confirm participation. From schools that agreed to participate, we obtained 
the names of mathematics teachers for eighth to twelfth graders. To a random selection of 
418 of those teachers, we mailed a copy of the questionnaire and a letter that described the 
project and asked them to complete and return the survey. No gifts or rewards were offered. 
We mailed the surveys on April 1 6 th and followed up with postcards two days later. 

Between May 8 th and May 23 rd , we contacted non-respondents, and, where necessary, mailed 
and faxed replacement surveys. We stopped data collection on June 23, 1997. Of the 418 
teachers, 296 returned completed surveys, 13 returned written refusals, and 88 did not 
respond. Twenty-one surveys were returned because the addressee did not teach 
mathematics or was currently on sabbatical. The completed questionnaires represent 71 
percent of all surveys sent out and 75 percent of the surveys that reached in-scope 
respondents. A complete breakout of these numbers is contained in Appendix B. 



Case Studies 



During April and May 1997, we conducted case studies in six areas across the 
country. To attain some measure of geographic dispersion yet limit travel costs, project 
coordinators identified six population areas (including two nearby) that would provide some 
diversity in district size and regional variation. Those areas were Baltimore City, 
Frederick, and Hagerstown, Maryland; Austin, Texas; Charleston, South Carolina; 
Milwaukee, Wisconsin; and Aberdeen, Bremerton, and Olympia, Washington. We sent an 
explanatory letter to the relevant district superintendents, enclosing information about the 
fieldtest and advising them we would be contacting the schools’ principals for permission to 
contact the mathematics teachers. From the principals of those public schools and the 

1 To obtain a sample of this size, we called 164 schools (87 public, 77 private). 
Twenty four schools did not participate because they had no staff member whose job 
description fit the study’s needs. 



private schools in those communities, we obtained approval to approach their eighth 
through twelfth grade mathematics teachers individually to recruit their participation in 
the case study. Through introductory letters and follow-up telephone calls to those teachers, 
we obtained commitments from fifty teachers. 2 Each teacher identified one particular 
course (“designated class”) for which they were willing to be observed once and to record 
classroom activities daily for four weeks. Together the classes covered the curriculum 
spectrum from eighth grade mathematics to Calculus. 3 Forty-one of the 50 volunteers 
ultimately completed the case study. 



There were five parts to the case study process: a mailed questionnaire, classroom 
observation, teacher interviews, daily class logs, and a second administration of the mail 
survey. 



Mail questionnaire: Before the case study began, participating case study 
teachers received the instructional practices questionnaire in the mail and 
completed it. They retained it until they could return it personally to the 
observer onsite. The questionnaire is in Appendix A. This was identical to 
the questionnaire mailed to teachers in the larger sample. 

Classroom observation: At the beginning of the case study, a researcher 
observed a class period in the teacher’s designated class, independently 
identifying and recording on a log form instructional objectives, classroom 
activities of the teacher and students, and availability and use of instructional 
materials. 

Teacher interview: Before the interview, teachers were asked to complete a 
classroom log form describing the class that had just been observed. During 
the following discussion, the observers compared their coding with the 
teacher’s coding of the same class, asking teachers to describe the 
instructional activities that had occurred, using the log form as a topic guide. 
Where teachers and observers differed in how they described and recorded a 
particular instructional activity, they each described what they observed and 
how it compared with their interpretation of the log form description. These 
interviews enhanced the teacher’s understanding about the meaning of key 
terms and the observers’ understanding of the conditions of instruction and 
the prevailing classroom routines and expectations. 



2 Of the fifty case study volunteers, 28 were in public schools and 22 were in private: 
25 were women, 25 were men. 

3 Of the 50 classes, 15 were Algebra I, 9 were Geometry, 8 were eighth grade Algebra 
or Pre-algebra, 7 were Algebra II, 5 were Calculus or pre-Calculus, and 6 were other types of 
mathematics courses. 



Daily logs of classroom activities: For the equivalent of four weeks of 
teaching (20 daily lessons or the appropriate number of block classes), case 
study teachers kept a daily log of their designated class. Using the log forms 
provided, teachers recorded a brief overview of the lesson content and 
structure, student learning objectives and amount of time spent on each, 
teacher and student activities, and materials used during that period. Each 
week, teachers mailed completed log forms to researchers. The case study 
teacher log form is in Appendix C. 

Mail questionnaire: At the conclusion of the data collection period, 
participating teachers received another instructional practices questionnaire 
in the mail to complete and return. Only 20 of the 4 1 case study teachers 
completed this second questionnaire, so the value of those questionnaires as a 
reliability measure is severely diminished. 



Analysis 

We used the mail survey data to calculate item response rates and item use rates, 
examine the distribution of responses across response options, and investigate potential 
threats to clarity for each survey item. 



The mail questionnaire item response rate is a measure of the 
proportion of all responses that is valid for each survey question; it shows the 
level of item clarity, precision, and respondent compliance with survey 
directions. It is calculated by dividing the total number of respondents that 
could have answered into the number that gave meaningful answers. Some 
questions were irrelevant for some respondents. The item response rates 
were generally high (97 percent or higher), ranging from 88.2 percent to 100 
percent; they are detailed in Appendix D. In the following discussion, we 
identify items with response rates less than 97 percent. 

The mail questionnaire item use rate is a measure of the number of times 
that survey respondents indicated they used each instructional objective, 
method, or material, and identifies the most frequently reported classroom 
conditions, content, instruction, and materials. It is calculated by dividing 
the number of respondents that could have answered into the number who 
gave meaningful answers. Item use rates are listed in Appendix D. 

The distribution of mail questionnaire responses is a measure of the 
appropriateness of the response options. The distribution of mail survey 
responses is in Appendix E. Where appropriate, discussions of the 
distributions are included below. 

Potential threats to clarity identify instances in which respondents may 
not have understood the terminology or may not have had sufficient 



information to answer the question. Where appropriate, they are reported 
below. 

We used case study data to assess the validity and reliability of teacher responses to 
survey items on student learning objectives, instructional activities, and material use and 
availability. This included analyzing teachers’ conceptual understanding of the student 
learning objectives and teacher and student lesson activities; the accuracy with which 
teachers estimated the amount of instructional time they and their students spend on 
specific activities; and the reliability of teachers’ responses to two questionnaires completed 
six weeks apart. 



We used the case study data to examine the distribution of responses across response 
options and investigate potential threats to clarity for each survey item. 



► The distribution of case study questionnaire responses is a measure of 
the appropriateness of the response options. The distribution of case study 
survey responses is in Appendix F. Where appropriate, discussions of those 
distributions are included below. 

We also used case study data to compare the teacher-completed classroom logs with 
observer-completed logs to make the following analyses: 



Case study teacher and observer use of classroom log items is a 

measure of the number of times during observations that teachers and/or 
observers indicated the teacher used each instructional objective or method, 
student activity, or materials. It identifies the most frequently used 
classroom instructional concepts and activities. We calculated item-use rates 
for teachers, observers, and both teachers and observers during the observed 
classes, and for teachers during the four week case study. Results are shown 
in Appendix G. 

The percent agreement between case study teachers and observers 
on occurrence of student learning objectives and instructional 
activities is a measure of the extent to which case study teachers and 
classroom observers both viewed a lesson as having included listed objectives 
and activities. It reveals among other things the extent to which teachers and 
observers demonstrated a similar understanding of the concept in question. 
Results are shown in column a of Appendix H. 

The percent of teacher/ observer nonagreement in which the teacher 
indicates the objective or activity occurred and the observer 
indicates it did not shows the direction of nonagreement between case 
study teachers and observers for each student learning objective or 
instructional activity. For all but two items, teachers viewed an occurrence as 



having happened although observers did not see it. Results are shown in 
column b of Appendix H. 

► The percent agreement between case study teachers and observers 
on length of time spent on student learning objectives and 
instructional activities measures the extent to which teachers and 
observers who agreed on occurrence, next agreed on the length of time it 
lasted. Results are shown in columns c and d of Appendix H. 

► The percent of teacher/observer nonagreement in which the teacher 
indicates the objective or activity occurred for a longer period of 
time than the observer did shows the direction of nonagreement between 
case study teachers and observers for each student learning objective or 
instructional activity. Results are shown in column e of Appendix H. 

Furthermore, we compared the teacher-completed questionnaire responses to the 
accumulated activity recordings from four weeks of classroom logs to make the following 
analyses: 

► The percent agreement between case study teacher questionnaire 
responses and logs on the occurrence of student learning objectives 
and instructional activities measures the consistency of teachers’ survey 
responses with their daily logs of classroom instructional objectives and 
activities maintained over a minimum of four weeks. It is a measure of the 
extent to which teachers’ survey responses reflect the same type of classroom 
activities as recorded on daily logs. Results are shown in columns a and b of 
Appendix I. 

► The percent of teacher questionnaire and teacher log nonagreement 
in which teachers underreport occurrence on the questionnaire, 
compared with log reports, further explores teacher survey and log 
nonagreement. It measures the percent of time in which the teachers’ survey 
responses indicate that an objective or activity occurs less frequently than 
their daily classroom logs indicate. Results are shown in column c of 
Appendix I. 

► The percent agreement between case study teacher questionnaire 
responses and logs on the duration of student learning objectives 
and instructional activities measures the consistency of teachers’ survey 
responses with their daily logs of classroom instructional objectives and 
activities maintained over a minimum of four weeks. Results are shown in 
columns d and e of Appendix I. 

► The percent of teacher questionnaire and teacher log nonagreement 
in which teachers underreport activity duration on the questionnaire 

further explores teacher survey and log nonagreement. It measures the 



percent of instances in which the teachers’ survey responses indicate that an 
objective or activity occurs for less time than their daily classroom logs 
indicate and helps to illuminate the direction of nonagreement between 
survey responses and daily classroom logs. Results are shown in column f of 
Appendix I. 

Finally, we used the case study data to compare the first questionnaire responses to 
the second questionnaire responses to assess the reliability of the items: 

► The percent agreement between case study teachers’ responses to 

the first and second questionnaires on the frequency and duration of 
classroom instructional practices measures the “test-retest” reliability of 
teachers’ survey responses completed approximately six weeks apart. Results 
are shown in Appendix J. 

Detailed descriptions of each analysis are included in Appendix K. 

When comparing a teacher’s recording on a daily log to the observer’s daily log, we 
assumed the observer to be the objective and disinterested party, and so measured the 
extent to which the teacher’s responses agreed; when comparing the accumulated responses 
from four weeks of the teacher’s logs to the teacher-completed survey responses, we used the 
logs as “truth.” To determine how close is close enough, we followed the lead of Burstein, 
McDonnell et. al. (1995) to establish standards for comparing two responses about the same 
event or set of events. For each item or subitem, we calculated (and report in the 
appendices) both the percentage of response pairs that were exactly identical and the 
percentage of those that were within one response category of each other. If 75 percent of 
the response pairs for each subitem are within one response category, we report the paired 
responses on that subitem as substantially consistent. 



The observer data used in these analyses are limited to the items included on the 
classroom log form, primarily items collecting information on student learning objectives, 
teacher actions, student activities, and material use and availability. For all comparisons 
involving the independent observer data, we removed subitems for which little or no use by 
observers or case study teachers resulted in too few cases to assess. We eliminated items 
receiving ten or fewer indications of use in the 82 records of the 4 1 classroom observations: 
two of nine student learning objectives, four of 12 teacher actions, six of 24 student 



activities, 4 and four each of eight teacher and student instructional materials. Those items 
were retained and considered, however, when we compared teacher questionnaire responses 
with teacher logs and when we compared responses to the first and second teacher 
questionnaires. A complete list of eliminated items is contained in Appendix L. 

To get an estimate of the accuracy with which teachers describe on a one-time 
questionnaire what they do throughout a semester without presenting them with too great a 
response burden, we collected log accounts of classes for the equivalent of four weeks of 
teaching. Then we compared teachers’ responses on the first questionnaire to the summed 
log accounts for the four weeks following. If the methods and items were apt and 
respondents’ memories were good, and if the four week period was representative of the 
activities throughout the semester, the frequency and duration of the instructional activities 
in the daily regimen recorded during the case study would be a relatively close match to the 
teachers’ responses on the questionnaire. 

First, to establish a common set of definitions with observed teachers, we compared 
teacher log forms with researcher observation forms for the same class. Using the 
information from the observed classes, we compared the learning objectives, teachers’ 
actions, and student activities recorded by the researcher with those recorded by the teacher 
for the same class. Differences in records generated discussion of log terms and eventually, 
greater confidence that what teachers communicated in their logs would be properly 
interpreted by the researchers. Second, to examine how well questionnaire responses 
reflected the frequency and duration of activities throughout a semester, we compared 
teachers’ four weeks of log data to their responses to the first questionnaire. We summed 
the log data to identify the frequencies with which teachers employed instructional 
techniques and the typical amount of time each was used, then compared those numbers 
with the appropriate questionnaire responses. Finally, to determine the reliability of the 
data, we examined the extent to which teachers’ responses to identical questions remained 



4 The two student learning objectives occurring infrequently enough during the 
observed classes to warrant removing them from analyses are (1) collect data (e.g., observe, 
measure, count) and (2) order, compare, estimate, and approximate. The infrequently 
occurring and therefore eliminated teacher actions are: (1) demonstrate a concept using 
three-dimensional tools, (2) provide remedial or enriching instruction to a pull-out group, (3) 
administer a test or quiz, and (4) demonstrate uses of technology. The student activities 
are: (1) write a report, (2) conduct lab or field work, (3) give oral reports, (4) work on week- 
long assignments, (5) participate in structured cooperative-learning activities, and (6) take a 
test or quiz. 



the same over time, using data from the first and second questionnaires completed 
approximately six weeks apart. 

When the data did not match (e.g., a teacher’s survey response indicated lectures 
once or twice a month, but the log data document lectures once or twice a week during the 
case study) , we also examined the direction of each mismatch to determine if teachers 
reported events happening more or less frequently (and for more or less time) on 
questionnaires than on logs. 

To evaluate consistency, we again used Burstein and McDonnell’s standards. If 75 
percent of the response pairs for each subitem were within one response category, we 
considered the paired responses on that subitem substantially consistent. 

When comparing responses from teacher questionnaires completed before and after 
the period of log-keeping, we used the responses from the later questionnaires to validate 
the first questionnaire. Only 20 of the 41 case study teachers completed the second 
questionnaire, so our analysis of teachers’ responses on the two surveys was limited to those 
20 sets, which supports limited generalizability. The case study teachers are their own 
comparison group, since we use their responses not to understand their teaching practices, 
per se, but to assess the reliability of their initial responses. 

To understand the characteristics of each item and obtain information on how each 
could be improved, we analyzed the items separately. This is different from the ways the 
final items are likely to be used by researchers or data analysts. Analysts using data from 
such items in their research generally combine multiple interrelated items to create more 
parsimonious composites describing underlying constructs of interest that no single item 
fully explains. Principal components analysis is one technique by which multiple variables 
are simplified and their inherent information amplified, “...instead of analyzing a large 
number of original variables with complex interrelationships, the investigator can analyze a 
small number of uncorrelated principal components.” 5 Combining items into one or more 
components also reduces the original variability contained within each original item, 
mitigating concern about the potential variability of individual items. 

Within each category of information collected, we report only those results that 
appear to be out of the norm or that provide unique and interesting information about the 



5 Computer-Aided Multivariate Analysis, Affifi and Clark, 1990, p. 372. 



quality, reliability, and future promise of the items fieldtested, focusing specifically on 
response rates, distribution of responses, and potential threats to clarity (primarily from the 
mail questionnaire data) and consistency and reliability (primarily from case study data). 



Results 

We report analysis results by areas of practice: teaching and learning conditions and 
classroom context; course content and emphasis; instruction; and instructional materials. 
Within each section we present information on response rates, distribution of responses, 
potential threats to clarity, and data about the consistency and reliability of self-report 
questionnaire responses. 



Items Assessing Teaching and Learning Conditions and Classroom Context 



Items in this section include those collecting information about the length and 
frequency of the class meetings, the number and grade levels of the students enrolled, their 
ability levels compared to others in the school, and the teacher’s assessment of student 
capability to learn the course material. Those items were: 

1 . Course title of designated class 

2. Class length and frequency 

3. Grade levels of students enrolled 

4. Percentage of students functioning at different ability levels (quintiles) 

5. Percentage of students capable of learning the required material 

6. Teacher influence on student grouping, student selection, and 
determining classroom discipline policies 

7. Teacher control in planning or teaching regarding decisions made (a) 
at the department, school, or district level and (b) at the classroom 
level 

14. Context factors (such as language diversity, social stresses, equipment 
shortages) that may require accommodation in instruction 

Response rates. Two items had relatively low response rates (about 88 to 93 
percent, compared with an overall 97 percent response rate). Item 3 asked respondents to 
record the number of students enrolled in the designated class who were at each grade level 
from sixth to twelfth. (An ungraded option was also offered.) Eighty-eight percent of all 
respondents provided appropriate responses. The most common inappropriate entry was a 
checkmark to indicate student grade level instead of the specific number of students. 



Item 7 also had relatively low response rates for some parts of the item. 

Respondents were asked to differentiate between two avenues of control over certain areas 
of planning and teaching: control “through participation in department, school, or district 
decisions” versus control “in my own classroom.” Response rates to the broader form of 
control ranged from 93.9 percent to 95.3 percent. Response rates to classroom control were 
uniformly higher, ranging from 97.6 percent to 99.3 percent. One or more of the following 
may have provoked the lower response rates: (1) the page layout (see Item 7 in Appendix A) 
required teachers to respond to the first segment by marking their responses to the left of 
the items (in contrast to responding to the more customary right side of the item), and some 
respondents may have failed to see that responses in that location were necessary; (2) some 
teachers may not have understood the type of control referenced; (3) some respondents may 
not have been able to differentiate conceptually between the two avenues of control; and (4) 
some teachers may have little opportunity or inclination to participate in decisions outside 
their classrooms and may have been dissuaded from describing the limited extent of their 
influence in that arena. In contrast, the response rates on the narrower classroom control 
may have been uniformly high because responses were marked to the right of the item, 
and/or because classroom control is a known issue and few teachers would be without 
opinion on their level of control within their own classroom. 

Item 4 on the relative academic ability of students had a response rate of 97 percent. 
This is noteworthy since the item required up to five numerical responses that together 
totaled 100 percent, a format considered difficult to manage. 

Distribution of responses. The distribution of responses to items on the 
conditions and context of instruction seemed appropriate. Some were flat, indicating wide 
and relatively equal variation; others were peaked in the middle showing fewer respondents 
on either extreme. We were especially interested in skewed distributions to understand the 
appropriateness of the response option scale. There were three such distributions in items 
on the conditions and context of learning. Item 7 on classroom control had two subitems 
each showing a large percentage of respondents with “complete control” (4 on a scale of 0 to 
4) over (d) selecting teaching techniques (80.0 percent) and (e) determining the amount of 
homework to be assigned (78.0 percent). Conversely, the percentage of respondents placing 
themselves on or toward the “no control” end of the spectrum (0,1, and 2 on a scale of 0 to 4) 
for those two parts was very low, totaling 3.0 percent (d) and 3.4 percent (e). Two 
distributions in Item 14 on the contextual factors influencing teachers’ decisions also show 
heavy skew. Asked about school safety, 80.4 percent of respondents perceived no threat (0 
on a scale of 0 to 3) to students’ personal safety (f) and 82.4 percent indicated no perceived 
threat to their own personal safety (o). 



Potential threats to clarity. Four comments indicated difficulty with Item 4, 
which asked teachers to indicate their students’ academic ability levels in mathematics. 

Two suggested that it is difficult to distinguish between ability and class performance. One ^ 

teacher wrote that responding involved “a lot of guessing,” and another said that the 
intervals (much above the school average; somewhat above, etc) were “too broad and 
arbitrary.” 

« 

Nine of the 297 mail respondents offered written comments about some aspect of 
Item 7, most of which concerned the forced distinction between the two avenues for control. 

Six of those comments suggested that it was difficult to indicate control “through 

participation in department, school, or district decisions” because the distinction between • 

that and control "in my own classroom” was fuzzy, vague, or unclear. Another respondent 

indicated that his level of control in the department was different from his control in the 

school and in the district and combining all three areas into a single response was not 

possible. £ 

Reliability of items on teaching and learning conditions and classroom 
context. Of the five items assessing teaching and learning conditions for which we have 
data 6 , three showed extremely strong agreement between case study teachers’ responses on ^ 

questionnaire 1 and on questionnaire 2. Those who completed both the first and second 
surveys gave the same answers each time on all 14 subitems on Item 14 (context factors), all 
fourteen subitems on Item 7 (teacher control); and two of the three subitems on Item 6 
(teacher influence). On the other hand, their early responses were different from later ones 
on Item 5 (percentage of students capable of learning the material) and all five subitems of ® 

Item 4 (student ability levels). There are several reasons why this might be so. The 
structure of both items is open-response and many more responses are possible and feasible 
compared to closed-ended questions. Rather than recalling a known percentage, it seems 
more probable that teachers respond to the item by estimating anew each time. % 

Items Assessing Course Content and Emphasis 

* 

Six survey items collected information about specific topics covered during the 
course, the level of emphasis teachers place on certain skills and concepts, student learning 



6 We do not have comparative information on course titles, class length and 
frequency, and student grade levels (Items 1, 2, 3). 
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objectives, assessment content, integration with other courses or subjects, and nonacademic 
time. Those items were: 

8. Content areas 

9. Lesson content emphasis 

10. Student learning objectives 

1 1 . Assessment content 

12. Interdisciplinary teaching 

16. Average nonacademic time 

Response rates. Only two items on course content and emphasis had low response 
rates. In contrast to open response Item 4, described above, Item 8 on specific areas of 
course content required written numerical responses in up to 15 spaces spread over two 
pages, totaling 100 percent. We knew this item might be difficult; indeed, it has evolved 
from several other studies efforts to make manageable the task of acquiring course content 
information. However, we decided it was worth trying. Ten percent of the surveys had 
unusable responses, mostly because the responses totaled something other than 100 
percent. The rate of usable responses may have been higher had we required teachers to 
total their answers; instead we inserted “100” in the total line to reaffirm the directions. 
Most respondents indicated that their designated course covered four to seven of the content 
areas listed. This suggests that the item can capture some of the content variation among 
courses with the same title, but did not advance understanding of how to estimate relative 
emphasis. 

The first portion of Item 9c requesting information on the frequency with which 
teachers emphasize broad mathematics concepts had a response rate of 95.9 percent— lower 
than adjacent items. There is no apparent formatting reason why 1 1 survey respondents 
left this subitem blank, more than any other segment of this item. Because it is sandwiched 
between two items with higher item response rates (97.3 and 98.3 percent), the lower rate 
seems to suggest difficulties with the conceptual understanding of the item: what are broad 
mathematics concepts and how does one emphasize them during instruction? 

Distribution of responses. Two items related to course content and emphasis 
have skewed distributions of responses. On Item 10b, 81.2 percent of mail respondents 
indicate they include understanding concepts, relationships, and theorems as a student 
learning objective more than one period per week, the highest frequency option. In Item 
11a, 88.2 percent of responding teachers say they include items that require students to 
compute answers on their student assessments twice or more per assessment, again the 
highest frequency option. 



Data from teacher observations and case studies document the frequent occurrence 
of teachers striving to help students understand concepts, relationships, and theorems (Item 
10b). During the 41 classroom observations, 92.7 percent of the teachers and observers 
indicated this student learning objective occurred. It was also the most frequent student 
learning objective during the four-week case studies, with teachers indicating that it was 
part of 83.6 percent of the lessons taught during that time. 

Potential threats to clarity. Written comments about items assessing course 
content and emphasis primarily concerned Item 8, asking teachers to indicate the time they 
spend in that class on 1 5 mathematics content areas. In addition to the expected (and 
obtained) comments that the list was too long and included too many categories, some 
teachers said the item was difficult to answer because (1) their teaching overlaps concepts 
across these defined areas; (2) they teach some concepts indirectly; or (3) they vary the 
content focus or emphasis over time in a way that makes it difficult to respond to the item. 
These teachers could not reconcile their understanding of their designated course’s 
instructional content to the construction of this item. 

Consistency of data on student learning objectives. Teachers’ daily 
instructional objectives for students are important because they provide the basis on which 
the teacher makes a host of other pedagogic decisions. Item 10 in the teacher questionnaire 
collects information on the frequency with which teachers employ nine general student 
learning objectives. 

Questionnaire responses from case study teachers indicated high use of four learning 
objectives (listed in order of frequency): (1) understand (e.g., concepts, relationships, 
theorems); (2) perform mathematical operations, execute algorithms, classify; (3) solve 
“story” problems with familiar structures, replicate proofs; and (4) memorize (e.g., facts, 
definitions, or formulae). The learning objectives used least were (in order of least use): (1) 
build and revise theories, develop proofs; (2) collect data (e.g., observe, measure, count); (3) 
order, compare, estimate, approximate; (4) interpret data (e.g., charts, graphs, tables); and 
(5) recognize and solve story problems with unfamiliar or complex structures. The same 
patterns of use occurred during observations: two of the learning objectives occurred 
infrequently enough that we did not collect sufficient data to analyze. Only five observers 
and/or case study teachers indicated that collecting data (Item 1 0c) occurred and only six 
indicated that ordering, comparing, estimating, and approximating (Item lOd) occurred 



i 

during those observed classes. 7 Because of this low use, we have not included these two 
subitems in analyses comparing teacher and observer responses. 

» 

For all 41 observed classes, we compared the student learning objectives observed by 
the researcher with those that the teacher indicated occurred during the observed class. 
Agreement indicates that both teacher and observer said the learning objective occurred; 

W nonagreement indicates that either the teacher reported it as happening and the observer 

did not, or the observer reported it as happening and the teacher did not. Over all of the 
learning objectives included in all the observed lessons, teachers and observers agreed 78.9 
percent of the time. The level of agreement between teachers and observers was greater 
£ than 75 percent for four of the seven objectives for which we report data: 90.2 percent 

agreement for performing mathematical operations was the highest and 65.9 for 
memorizing was the lowest. 

P As these two extremes suggest, the level of teacher/observer agreement seemed to 

vary by the degree to which the student learning objective was observable by a classroom 
visitor or was explicitly stated to the class or to the observer by the teacher. For example, it 
was usually clear when students were performing mathematical operations, but often 
difficult to observe that students were expected to be memorizing. The learning objectives 
^ with the lowest rates of agreement (memorizing facts, definitions, or formulae; recognizing 

and solving story problems with unfamiliar structures; building and revising theories or 
developing proofs) are less visible and simply more difficult to detect. This suggests that 
learning objectives not observed by classroom researchers may indeed have occurred, but 
® were simply not observed. 

We examined the circumstances of such nonagreement between teachers and 
observers to identify whether teachers or observers were more likely to report that a 
£ particular learning objective had occurred. Overall and for all learning objectives, teachers 

were more likely than observers to report that a learning objective was part of the observed 
lesson. For five of the objectives, disagreements occurred as often when teachers noted an 
event that observers missed as the reverse. For the other two, teachers reported using an 
^ objective when researchers reported they had not observed it. 



7 Discussions with case study teachers suggest that some teachers may consider 
(lOd) ordering, comparing, estimating, and approximating not as learning objectives but as 
basic skills that are systematically taught (and perhaps little used in class). 
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We also examined the level of agreement between teachers and observers on the 
minutes spent on each learning objective. That information indicates a high level of 
agreement between teachers and observers on the minutes allocated toward each student 
learning objective for a given class period. Agreement within one response category 
between teachers and observers was at least 75 percent for each of the seven objectives, and 
was 100 percent for two objectives. There was no pattern to the direction of nonagreement 
in the other five objectives, with teachers indicating either more or less time allocated than 
observers noted. 

To learn more about the quality of questionnaire items assessing student learning 
objectives, we measured the consistency of teachers’ survey responses of the frequency with 
which they employed each of the nine student learning objectives with their daily log entries 
over four weeks. For three of those objectives, the level of agreement between logs and 
questionnaires was greater than 75 percent. Those with a high level of agreement were 
understanding concepts, relationships, and theorems (10b); performing mathematical 
operations (lOe); and building and revising theories (lOi). The two most commonly used 
objectives (10 b, lOe) had two of the highest levels of agreement. 

Examining the directions of the nonagreement between a teacher’s questionnaire 
responses and log entries, we discovered that the majority of teachers underreported on the 
questionnaire, compared to the portraits of instruction obtained from their daily logs. Fully 
71.2 percent of all nonagreeing responses underreported on the questionnaire the frequency 
with which student learning objectives occurred in their classrooms. This was also true for 
individual learning objectives: the percentage of teachers underreporting on the survey was 
greater than 50 percent in eight of the nine student learning objectives. 

Reliability of items on course content and emphasis. Of the five items 8 
assessing course content and emphasis for which we have information to compare teachers’ 
responses on questionnaire 1 and on questionnaire 2, four items showed extremely strong 
agreement. All eight subitems on Item 9 (lesson content emphasis), all nine subitems on 
Item 10 (student learning objectives), 1 1 of the 12 subitems on Item 1 1 (assessment 
content), and all of the three subitems on Item 12 (interdisciplinary teaching) had high 
levels of agreement. Furthermore, the eight responses in Item 9b on the typical length of 
time spent per class period emphasizing each dimension all had more than 75 percent 
agreement within one response category. On the other hand, on Item 16, which asked 

8 We did not compare teacher responses on Item 8 (content areas) that required up to 
15 open-ended responses totaling 100 percent. 



teachers to estimate the amount of time spent on nonacademic tasks, answers varied a lot 
from questionnaire 1 to questionnaire 2. Similar to Items 4 and 5 discussed under teaching 
and learning conditions, the explanation may simply be that the answers are open-ended, 
requiring a percentage response that teachers may newly construct each time. 



Items Assessing Instruction 

Three multi-part items collect information about the frequency and duration of 
teacher instructional actions, student activities, and the ways in which students 
demonstrate proficiencies. Those items were: 

13. T eacher actions 

15. Student activities 

17. Student demonstration of mathematics competencies 

Response rates. Two related items on tests and quizzes had relatively lower 
response rates than the 97 percent average. On Item 13i, the second portion asks teachers 
who administer tests or quizzes to indicate the amount of time the assessments typically 
take. The response rate was 95.9 percent for this item. Of the 14 unusable responses, nine 
were missing and five were inappropriate (two responses were circled rather than one). On 
Item 15w the second portion had a similar response rate of 96.6 percent. Related to 13i on 
teacher time. Item 15w asks for the amount of time students spend taking tests, quizzes, or 
other assessments. Of the 1 1 unusable responses, five were missing and six were 
inappropriate. 

Distribution of responses. There are several skewed response patterns on the 
mail questionnaire items assessing instructional practices. The following two items show a 
high percentage of teachers who indicate they do the following more than once a week (the 
most frequent response option) : 

% Giving 

Sub Highest 

Item Frequency Sub Item Content 

13a 76.9 percent lecture, perhaps occasionally soliciting brief student input or 

using the board or overhead projector to highlight a key term or 
present an outline 

13b 70.7 percent demonstrate a concept, using two-dimensional graphics such as 

drawings on the board, overhead projector, or computer 



The prevalence and frequency of lecturing as indicated in the mail survey responses 
above is corroborated by the case study logs that indicate that lecturing was the most 
common teacher instructional activity, occurring at least briefly during 80.9 percent of the 
700 individual class sessions covered by log-keeping. 

Similarly, four items on student activities have skewed responses suggesting high 
usage. The following shows the percentage of respondents who indicated on questionnaires 
that their students do the following more than once a week: 

% Giving 

Sub Highest 

Item Frequency Sub Item Content 

15a 83.4 percent listen to or observe teacher presentations 

15h 85.1 percent correct or review previous day’s homework 

Our interviews with case study teachers suggest that these percentages are reflective 
of practice. Observations offered mixed evidence. During the observed classes, all teachers 
and observers indicated that students listened to or observed teacher presentations; 54 
percent indicated that students worked on their previous day’s homework for at least a 
small portion of class. It is difficult to explain the differences on the homework item, since 
other experiences suggest that it is a common occurrence and no format differences exist 
that would affect responses. 

Potential threats to clarity. Written comments about the three items assessing 
instructional methods on the returned mail surveys indicate that some teachers see tests 
and quizzes as substantively different events and could more easily respond to Item 13i on 
frequency of occurrence if they were two separate items. Another saw redundancy between 
13f (provide individual or small group tutoring as needed during individual seatwork or 
small group activities involving everyone in class) and 131 (set up and monitor or supervise 
cooperative learning activities), “since cooperative learning takes place in small groups.” 

Consistency of data on teacher instructional actions. The items on the 
instructional activities of the students and teacher are the core of the questionnaire. They 
seek to provide information about specific instructional events that often occur in secondary 
mathematics classrooms. Item 1 3 collects information on the frequency and duration of a 
range of activities in which teachers may engage while teaching. 



Questionnaire responses from case study teachers identify three teacher activities 
that receive high use: lecture, demonstrate a concept using two dimensional graphics, and 
provide individual or small group tutoring. Other teacher activities are little used: 
demonstrate a concept using three dimensional tools, provide remedial or enriching 
instruction to a pull-out group, and administer a test or quiz. Similar patterns of use and 
nonuse occurred during observations and we removed those three subitems plus 
“demonstrate uses of technology” from analyses comparing teacher and observer responses 
because of insufficient data 9 . 

To determine the similarity with which teachers and independent observers 
interpreted the teacher activities in the classroom and on the survey, we compared the log 
entries of case study teachers and observers using a similar process to that described above 
for student learning objectives. These data indicate a high level of agreement between 
teachers and observers on the occurrence and duration of teacher instructional activities. 
During all of the observed lessons, teachers and observers agreed on 84.6 percent of all 
teacher activities. In seven of the eight activities, the level of agreement between teachers 
and observers about whether the activity occurred was 75 percent or greater. 

The highest rates of agreement between teachers and observers were for lecturing 
(97.6 percent) and providing individual or small group tutoring (95 percent). Observers 
most often disagreed with teachers on whether or not teachers had “stimulate (d) student 
discussions of approaches to solving problems, explanations of their mathematical thinking, 
or open-ended questions.” For example, although case study teachers might say after class 
that they had stimulated student discussions of alternative approaches, observers did not 
classify the observed student discourse in the same way. Consequently, that item had the 
lowest agreement rate: 55 percent. 

The same pattern appeared in the other instances of nonagreement between teacher 
and observer: where there was nonagreement on whether an activity occurred, teachers 



9 During the case study observations, the teacher activity of demonstrating uses of 
technology was recorded on the 82 observation forms only nine times, too few to provide 
reliable information and the data are not included. Interestingly, though, of those nine 
times, eight were recorded by teachers and only one recorded by an observer. Even for such 
a small number of occurrences, this seems overwhelmingly lopsided. At best the ratio 
suggests seven “unanswered” statements in which teachers said they were demonstrating 
technology to students. Although no further information from observers is available to 
enlighten those events, it is possible that the instances involved calculator uses or 
demonstrations, rather than other forms of “technology” use. 
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were more likely to report that it did happen than were observers. In 94 percent of the 

instances in which teachers and observers had different entries on the observation form, 

teachers indicated the activity had occurred and observers indicated they had not seen it. ® 

We found a high level of agreement between teachers and observers on the minutes 
spent on each teacher instructional activity that occurred during the observed class period. 

Agreement within one response category between teachers and observers was at least 75 ^ 

percent for seven of the eight teacher activities, and 100 percent for five of them. In 

instances of nonagreement with observers on the length of time allocated to each teacher 

instructional activity, there was no systematic pattern to teacher responses. Teachers were 

just as likely to underestimate the amount of time spent as they were to overestimate, in 9 

comparison with observers’ estimates. 

We also measured the consistency of teachers’ survey responses about the frequency 
of instructional activities with their daily log entries during the four week case study. The g 

item levels of agreement averaged 61.9 percent agreement across all logs and 
questionnaires. Lecturing had the highest level of agreement at 80.5 percent, while two 
items had rates less than 50 percent: 28.2 percent for working on administrative tasks while 
students worked on assignments, and 48.7 percent for leading students in recitation, drills, 
or question-and-answer sessions. ^ 

Investigating the direction of the nonagreement between logs and questionnaire 
responses suggests that teachers were more likely to underreport the occurrence of their 
activities when responding to the questionnaire. For eight of the twelve activities, more ^ 

than 50 percent of those respondents whose two records did not agree underreported the 
occurrence on the questionnaire. This suggests that the following activities may occur more 
frequently than survey results indicate: recitation and drill, administrative tasks, small- 
group tutoring, student discussion on multiple approaches to problem solving, and student- 9 

led discussions. Teachers were more likely to overreport on the questionnaire the frequency 
with which they demonstrate concepts using two- or three-dimensional graphics and set up 
and monitor cooperative learning activities. 

t 

Of the twelve teacher activities, the two with the lowest agreement rates between 
logs and questionnaires also had the highest rates of underreporting on the questionnaires. 

For example, 90 percent of the case study teachers whose questionnaire responses (about 
the frequency with which they worked on administrative tasks) did not agree with their log 
records underreported that activity. Their log records indicate that they actually worked on 
administrative tasks more frequently than they indicated on the questionnaire. 
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There was substantial agreement between teacher questionnaire responses and 
teacher logs summed over the case study period on the duration of the teacher’s 
instructional activities. In all but two teacher activities, the questionnaire responses of at 
least 50 percent of the case study teachers closely matched log records, and for six of the 
twelve activities those records agreed for more than 75 percent of teachers. The highest 
levels of agreement, ranging from 82.9 percent to 95.1 percent, were recorded for lecturing, 
small group tutoring, administering a test, leading student discussion of multiple 
approaches, demonstrating concepts with two-dimensional graphics, and leading recitation 
and drills, while the least agreement was associated with demonstrating concepts using 
three-dimensional tools and providing remedial instruction to pull-out groups. Where 
teachers’ questionnaire responses about the amount of time spent on instructional activities 
were not supported by their log diaries, teachers generally reported more time on the 
survey. In ten of the twelve teacher activities, a higher percentage of teachers overreported 
allocated time than underreported. The majority of teachers underreported on the 
questionnaire— relative to the logs— the amount of time spent on two teacher activities: 
stimulating student discussions of multiple approaches and leading students in recitation or 
drill exercises. Teachers spent more time on those activities during the four case study 
weeks than they indicated on the questionnaire. 

It may well be that social desirability acts as a bias to create some of these patterns. 
When removed from the immediate experience of teaching, respondents tended to remember 
using more practices that are consistent with well promoted NCTM standards and fewer 
traditional practices. Likewise, they tend to down-play time spent on “administrivia.” On 
logs, the immediacy of experience corrects for these tendencies. 

Consistency of data on student learning activities. Item 15 provides 
information on the frequency and duration of specific student instructional events. 
Responses from the case study questionnaires show five prominent student activities that 
occur frequently: listen to or observe the teacher, correct or review previous day’s 
homework, work individually on exercises, work on assignments due as homework the next 
day, and wait for completion of nonacademic tasks. Similarly, questionnaire responses 
indicate that several other student activities happen seldom: writing a report, conducting 
lab or field work, giving oral reports, and working on week-long assignments. These and 
two other events (structured cooperative learning activities and taking tests or quizzes) 
occurred too rarely during the case study observations to include in our analyses comparing 
teacher and observer responses. 



Comparing case study teachers’ and independent observers’ recordings of student 
classroom activities indicates a fairly solid level of agreement between teachers and 
observers on whether or not specific classroom activities occurred. Teachers and observers 
agreed on 82.4 percent of all student activities during the observed lessons, and the level of 
agreement between teachers and observers about whether the activity occurred was 75 
percent or greater in 13 of the 18 student activities. 

Student activities with high rates of agreement in observer and teacher records are: 
listen to the teacher (100 percent); work individually on exercises, etc. (92.7 percent); work 
in small groups (92.7 percent); and work on assignments due the next day (85.4 percent). 
Student activities with the lowest agreement between teacher and observer were the 
following: participate in whole-class discussion (56.1 percent); practice or drill on 
computational skills (63.4 percent); wait for completion of nonacademic tasks, e.g., 
attendance, homework collection, behavior management, etc. (70.7 percent); solve problems 
for which there are several appropriate answers or approaches (70.7 percent); and correct or 
review previous day’s homework (73.2 percent). 

Teachers and observers reported differences of opinion on what constituted the first 
three activities: whole-class discussions; problems for which there are several appropriate 
answers or approaches; and whether students were waiting or were gainfully employed. In 
the majority of nonagreements, teachers indicated that student discussions involved the 
whole class, while observers were more likely to say that only a few students were actively 
involved. Teachers more often said after class that problems had several approaches, when 
observers saw a single one emphasized. 

When teachers and observers did not agree on whether students responded orally to 
open-ended questions or explained to the whole class solutions developed individually or in 
small groups, observers were more likely than teachers to indicate that those activities 
occurred. In all other instances of nonagreement, teachers were more likely than observers 
to have indicated the event occurred. 

There was a high level of agreement— from 85.7 to 100 percent— between teachers 
and observers on the length of time each student activity occurred during the observed class. 
There was no pattern in the direction of nonagreement: teachers over- and under-estimated 
nearly equally, relative to observers’ estimates. 

We also measured the consistency of teachers’ survey responses about the frequency 
of student learning activities with their summed daily log entries from the four-week case 



study. The levels of agreement ranged from 46.2 to 100 percent, and averaged 69.1 percent 
across all responses. Agreement was greater than 75 percent for 1 1 of the 24 activities, and 
the highest agreement was for lecturing. Two items had agreement rates less than 50 
percent: 46.2 percent for practice or drill on computational skills and 47.5 percent for using 
supplementary printed materials. 

For those questionnaire responses about the frequency of student activities that were 
not confirmed by teacher classroom log form tallies, we also examined the direction of the 
nonagreeing responses. For each of 13 student activities, the majority of the teachers whose 
log accounts did not reflect their questionnaire responses underreported on questionnaires 
the frequency with which their students participated in those instructional activities. In 
only four of the student activities listed did more teachers overreport the frequency with 
which their students engaged in those activities: (1) correct or review previous day’s 
homework: (2) respond orally to questions testing recall: (3) respond orally to open-ended 
questions: and (4) wait for completion of nonacademic tasks. 

Similarly, we used the case study log forms to evaluate the accuracy with which 
teachers estimated the length of time students spent on certain learning activities. For 1 1 
activities, 75 percent or more of teachers’ estimates of time were accurate within one 
response category of the times indicated on their classroom logs. At least 90 percent of 
teachers’ responses were accurate at that level for eight of those activities. For three 
activities, the majority of teachers whose questionnaire responses did not accurately reflect 
the duration indicated on their log forms were more likely to overreport on questionnaires 
the amount of time they allocated to those student activities. The activities were: giving 
oral reports, using supplementary printed materials, and engaging in on-task discussion 
with other students. This suggests that these activities may occur for shorter periods of 
time than questionnaire responses might indicate, in any case, they occupied less time in 
the log period than estimated for the semester. 

Teachers may be more accurate in reflecting the time spent on the student activities 
that they use more often and less accurate estimating time spent on activities they use less 
frequently. For example, case study teachers indicated that students listened to them 
lecture during 90.6 of the 700 lessons for which we have logs, and the questionnaire 
responses on time per typical use accurately reflected the duration of those events within 
one response category for 95.1 percent of the teachers. On the other hand, case study 
teachers had students use textbooks for reading or reference during only 27.7 percent of the 
classes and only 48.8 percent of the questionnaire responses on time per typical use 
accurately reflected the duration of those events within one response category. 



Reliability of items on instruction. Ten of the twelve subitems in Item 13 
assessing the frequency with which teachers employ certain instructional techniques 
showed high correspondence between teachers’ responses on the first and second 
questionnaire. Additionally, all 12 of the responses in Item 13 on the typical length of time 
spent per class period emphasizing each dimension had more than 75 percent agreement 
within one response category. Two items had low rates of agreement between the first and 
second questionnaires: leading students in recitation and drills and teacher time spent 
working on administrative tasks. 

Of the 24 subitems in Item 15 assessing the frequency with which teachers have 
students engage in particular learning activities, 22 showed high correspondence between 
teachers’ responses on the two questionnaires. All 24 time per typical use items had a 
greater than 75 percent agreement within one response category. The two items with low 
rates of agreement between questionnaire 1 and questionnaire 2 were (a) solving problems 
with more than one appropriate solution and (b) practice or drill on computational skills. 

Nine of the ten subitems in Item 17 on student demonstration of mathematics 
competencies had high reliability between the two case study teacher questionnaires, while 
only 70.6 percent of responding case study teachers were consistent in their two responses 
to subitem 17i, generalizing from patterns or examples. This may suggest that teachers are 
less certain about how frequently they have students demonstrate this particular 
competency. 



Items Assessing Availability and Use of Instructional Materials 

Two items collected information about the availability of various materials and 
equipment classroom instruction. Those items were: 

18. Teacher materials 

19. Student materials 

Response rates. Only a single subitem on the availability and use of materials had 
a relatively low mail questionnaire response rate. Item 19f on the availability to students of 
film or videotape had an item response rate of 95.6 percent, primarily due to missing 
responses. The response rate on the parallel teacher Item (18f) was 97.3 percent. 



Distribution of responses. Questionnaire responses to a number of items in this 
section are skewed, reflecting substantial availability of certain materials for both teachers 
and students. More than 70 percent of all respondents say that the following materials are 
readily available to them and to their students: a board; graph paper; protractors, rulers, or 
compasses; appropriate calculator; and an overhead projector. Eighty-four percent of 
teachers report using the board more than once a week. Seventy-three percent of teachers 
report that they and their students use an appropriate calculator more than once a week. 
The numbers seem to reflect accurately the availability and use of these particular 
materials in the classrooms we observed. 

Consistency of data on availability and use of instructional materials. 

Responses from the case study questionnaires indicate that teachers routinely and 
frequently use three types of materials in class: the board, an appropriate calculator, and to 
a lesser extent, an overhead projector. Responses also indicate relatively rare use by 
teachers of film or videotape, computers, graph paper, protractors or rulers, and 
manipulatives or models. Based on lack of use during the observed classes, we eliminated 
each of these except manipulatives and models from our analyses comparing teacher and 
observer responses. Similarly, case study questionnaire data indicate that students make 
frequent use of calculators, and to a lesser extent, the board and graph paper, and rare use 
of film or videotape, computers, manipulatives or models, and overhead projectors. 
Accordingly, we removed all four from the analyses. 

Case study information is similar to that from questionnaires on the availability of 
the materials identified above. We compared the case study teachers’ and observers’ 
responses on materials to determine their level of agreement. The board, a calculator, 
overhead projector, and manipulatives were used by the teacher in nearly every class, while 
they rarely used graph paper, protractors, rulers, compasses, film or videotapes, or 
computers. Student use of materials was similarly narrow, but included graph paper, 
protractors, rulers, or compasses in addition to an appropriate calculator and the board. 

The teacher and observer records indicate a high level of agreement on the use of all 
materials by teachers and students. At rates greater than 80 percent for all types of 
materials, teachers and observers agreed on whether or not teachers and students used the 
materials. In the relatively few cases of nonagreement, teachers were more likely them 
observers to say that certain materials had been used during the lesson. 

Teachers and observers also had high levels of agreement on the length of time with 
which those materials were used during the observed lessons. With one exception, the two 



groups agreed on the length of time within one response category more than 7 5 percent of 
the time. In 29 percent of the observations, teachers and observers disagreed over the 
length of time that teachers used a calculator during class, with most teachers indicating 
they used the calculator for a longer period of time than observers noted. Later discussions 
between teachers and observers suggested that observers took a more literal view of the 
amount of time that teachers used a calculator, compared to the teacher’s view of continual, 
but sporadic use throughout the period. 

To measure the reliability of questionnaire items assessing the frequency with which 
teachers and students used instructional materials, we compared case study teacher 
questionnaire responses with that teacher’s log entries. For teacher use of the materials, 
the levels of agreement ranged from 62.5 percent to 87.2 percent and the single item with a 
rate of agreement less than 75 percent was for calculator use. When a teacher’s 
questionnaire response and log entries did not agree, teachers tended to overreport on the 
questionnaire their use of the board and calculator, and underreport their use of an 
overhead projector and manipulatives. 

The levels of agreement between teacher questionnaire responses and log entries for 
student use of materials ranged from 52.6 percent to 75.0 percent and was less than 75 
percent for three materials: the board; graph paper; and protractors, rulers, and compasses. 
More teachers tended to underreport on the questionnaire student use of the board; graph 
paper; and protractors, rulers, and compasses, and overreport student use of calculators. 

Reliability of items on instructional resources. All of the eight subitems on 
teacher materials and on student materials showed very high agreement between the first 
and second questionnaires on both availability and frequency of use. 



Conclusions 



This fieldtest used mail survey responses and data from case studies with observers 
and teacher-completed classroom logs to assess the validity and reliability of teacher 
responses to survey items about the mathematics instructional practices commonly used in 
eighth to twelfth grades. We report conclusions drawn from that information about the 
reliability and accuracy of the data collected through the fieldtested items, the potential for 
large-scale use of the items and methods, and their applicability to other high-interest 
content areas. 



Accuracy of Teachers’ Descriptions of Daily Teaching 



Case study teachers ’ questionnaire responses on the student learning 
objectives, teacher actions, and student activities occurring and the 
instructional materials being used in the teachers’ designated classes were 
substantially confirmed by independent observers in 32 of 41 items on the 
teacher questionnaire. Observers substantially agreed with case study teacher 
responses to seven of the eight items describing teachers’ instructional actions, 13 of the 18 
items describing student activities, and four of the seven items detailing student learning 
objectives. Classroom observations supported teacher responses to four items on teacher 
materials and to four items on student materials. Where there was nonagreement between 
case study teachers and observers on whether or not certain student learning objectives and 
instructional practices occurred, case study teachers tended to indicate more often that an 
event had occurred when an observer did not see it. This was also true for the 
nonagreements occurring in 15 of the 18 student activities. 

The extent to which case study teachers and observers agreed on whether or not an 
event occurred depended to a large extent on the degree to which the activity was clearly 
defined, discrete and distinct from the other activities, and observable by an outsider. Of all 
the teacher activities included in Item 13, for example, subitem j (“stimulate student 
discussions of approaches to solving problems, explanations of their mathematical thinking, 
or open-ended questions”) was arguably the least defined and most difficult to observe. 

Also, noteworthy in this respect are the terms “student discussions” and “mathematical 
thinking.” Teachers and observers alike expressed difficulty (1) deciding whether one or two 
students describing their solutions to a problem, for example, constituted a “student 
discussion;” (2) determining how many students were needed and for what period of time 
before classroom discourse could be labeled a student discussion; or (3) deciding when a 
description of the process a student followed to arrive at an answer became an “explanation 
of their mathematical thinking.” Not surprisingly, then, the highest reliabilities between 
case study teachers and observers occurred with items listing observable, discrete events. 
Teacher responses to items on unobservable events may be no less valid, but their reliability 
is much harder to document through observation. 

Case study teachers’ questionnaire responses on the length of time during 
which (1) student learning objectives were taught, (2) teachers and students 
engaged in activities, and (3) student and teacher materials were used in the 
teachers’ designated classes agreed with descriptions of independent observers 
in 40 of 41 items on the teacher questionnaire. Observers substantially agreed with 



case study teachers’ description of the durations of each of the seven items detailing student 
learning objectives, the eight items describing teachers’ instructional actions, and the 18 
student activities, and on three of the four uses of teacher materials. Observers also 
validated teacher responses on the length of time that three teacher material items were 
used and on the length of time students used the materials identified in the items on 
student materials. Teachers were most consistent in estimating the amount of time they 
spent on those activities they used frequently. This may indicate that teachers are 
consistent in the amount of time they spend on these frequently-used activities and 
therefore better able to estimate time accurately. Where there was no agreement on how 
long a particular objective or activity occurred, teachers were just as likely as observers to 
underestimate an event’s elapsed time. This was true for all five types of items. 



Accuracy of Teachers’ Descriptions of Teaching Over a Semester 

Case study teachers’ questionnaire responses on the student learning 
objectives, teacher and student activities, and the instructional materials being 
used in the designated classes were not supported by data they collected in four 
weeks of classroom logs in 37 of 61 items on the teacher questionnaire. Where 
the teachers’ questionnaire responses did not agree with their log recordings on whether an 
objective or activity had occurred, teachers were more likely to underreport that an event 
had occurred. 

This suggests that some teacher and student activities may actually occur more 
frequently than survey results indicate. This is as true for such methods as recitation and 
drill as for student discussion of multiple approaches to problem solving. Case study 
teachers in the sample, however, tended to report more on the teacher questionnaire than 
on the logs occurrence of three NCTM recommended instructional activities: demonstrating 
concepts using two- or three-dimensional graphics, demonstrating uses of technology, and 
using cooperative learning activities. This suggests the possibility of bias in the direction of 
social desirability. 

Case study teachers’ questionnaire responses on the duration of events in 
the teachers’ designated classes were substantially validated by classroom logs 
in 17 of 36 items on the teacher questionnaire. Case study teachers’ description of 
the durations of 6 of the 12 items detailing teacher instructional activities, and on 1 1 of the 
24 items describing student activities were borne out by the event information recorded on 
the classroom log forms. Where there was no agreement on how long a particular objective 



or activity occurred, teachers tended to overreport on questionnaires the duration of an 
event. This was true for most teacher and student activities. 

Some items may have low agreement between teacher and observers 
because classroom observation is limited in its capacity to capture certain 
elements of classroom instruction. Using classroom observations to validate self 
reports of teacher practice is limited, quite simply, by the degree to which the report or 
activity can be detected and observed by someone else. Some instructional practice items 
lend themselves more readily to observation than others. Observers in this fieldtest found it 
particularly difficult to identify certain student learning objectives. This could happen for a 
number of reasons. Although the teacher may have intended a particular objective to be 
part of a lesson, the actualization of that objective may not have occurred or may not have 
been clear to the observer (and perhaps the students) . We could not determine the 
proportion of item inconsistency caused by misleading item construction compared to that 
resulting from ineffective teaching. 

The validation of other items may be affected by the respondent’s 
perception of the social desirability of certain actions being described. Subtle or 
not-so subtle pressures may influence teachers’ responses to questionnaire items that may 
be politically sensitive, especially those perceived to be particularly old fashioned or more or 
less desirable in a climate of reform. Teacher responses to two items about traditional 
practices, for example, may reflect the influence of social desirability: (13d) lead students in 
recitation, drills, or question-and-answer sessions, and (13h) work on administrative tasks, 
such as recordkeeping, while students work on assignments. Each has reasonably high use 
on the observation logs, high teacher/observer agreement on that use, and substantial use 
during the case study period, but low correspondence between questionnaire responses and 
the summed daily logs and between the first and second questionnaires. 
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students work on assignments 



Reliability of the Data 



Case study teachers’ responses were substantially the same on 89 percent 
of the questionnaire items on questionnaires completed six weeks apart. Half 
of the items that differed were open-ended questions requiring responses in a 
percent format. Case study teachers’ responses on two questionnaires completed five to 
six weeks apart substantially validated each other in 1 10 of the 124 items describing the 
context, teaching and learning conditions, instructional activities, and instructional 
materials, including 57 of the 61 items that were the primary focus of the case study. 
Teacher responses to a second questionnaire completed later substantially agreed with their 
first responses on 23 of 30 items about the teaching and learning context, on 31 of 33 items 
about course content, on 40 of 45 items on instruction, and on all 16 items about 
instructional materials. 



Potential for Large-scale Use 

The instructional practice items that hold the most promise for large- 
scale use are those that have a clear and understandable definition and clearly 
defined limits. The challenges of creating self-report instructional practice items for 
large-scale use appears to be similar to the challenges that face all data collections: the best 
items are grounded in a clear understanding of the instructional practices in question, have 
distinct definitional boundaries, and have been honed through fieldtests with the broadest 



possible range of potential respondents. The best items anticipate and prevent the 
ambiguities that might occur, for example, when teachers from one region of the country 
refer to a particular instructional practice with a label different from that used by teachers 
elsewhere, when new instructional techniques are in the midst of development and the 
knowledge and understanding of the techniques is not universal, or when the meaning of a 
common term (such as cooperative learning) takes on a new and narrower definition. 



Parts of some items were too complex, too inclusive, or required teachers 
to make too-difficult distinctions in their practices, as indicated by low rates of 
usable responses on the mail survey. 



► Low response rates to parts of Item 7 on teacher control over instruction (a) in 
their own classroom, and (b) through participation in decisions outside their 
classroom suggests problems with the item. The complex question required 
teachers to differentiate the extent of their control by area and by process. 

The layout of the item may also have contributed to a low usable response 
rate by requiring two responses on each line: one response in front of the 
item and one response following the item. Mail respondents had little 
problem with responses to the right of the subitems on control within the 
classroom, but there were fewer usable responses to left of each subitem. 

► The low response rate to Item 8 on course content suggests that responding 
appropriately to a single open-ended item with 15 blanks over two pages is 
daunting even to mathematics teachers. Coupled with the item’s conceptual 
requirements and printed density, results from this fieldtest suggest that 
collecting information on course content via such an open-ended item is too 
complex, time-consuming, and challenging. 

► Inappropriate responses on Item 13i assessing the frequency with which 
teachers use tests and quizzes in their instructional repertoire suggests the 
question is too inclusive. By combining tests and quizzes in a single item, we 
were in effect asking teachers to ignore differences in two practices they view 
discretely. 

High frequencies of use on some survey items suggest that either some 
items may need to be more narrowly defined or some practices are very 
pervasive in mathematics classes. 



95.2 percent of mail questionnaire respondents and 97.6 percent of case study 
questionnaire respondents indicated they use the student learning objective 
understanding concepts, relationships, theorems (Item 10b) once a week or 
more. This high rate of use could suggest that the phrase is so general that it 
is devoid of recognizable meaning for mathematics teachers used to working 
with specific skills and techniques. 



► Teacher responses to Item 11a indicate that 95.3 percent of mail respondents 
and 97.5 percent of case study teachers use items that require students to 
compute answers an average of once or twice per assessment. Similarly, two 
other items have responses that indicate that 90 percent or more of the 
respondents to both surveys employ those student activities more than once 
per week. 

Response rates for items requiring two responses per line were equal to 
those requiring a single response. The responses to duration questions in part b 
provide additional information about activities employed with similar frequencies. The 
ability of respondents to handle two-part items requiring two responses per line had been a 
question in the early design phase, and results here confirm earlier fieldtest results that 
teachers can respond appropriately to such items. This is important since data collected 
through the two-part questions (for example on both the frequency and the length of time 
that teachers employ specific instructional techniques) provide useful information about 
course emphasis on topics. For example, similar high percentages of respondents to the 
mail questionnaire indicated their students listen to or observe teacher presentations (15a) 
and correct or review the previous day’s homework (15h) more than once per week. 

However, students typically review their homework for less than ten minutes but listen to 
their teacher for a longer time— up to half of the period. Without that distinction in event 
duration, those two activities would appear equally important. 

The frequency response options and time per-typical-use options collect 
data about traditional school schedules and block schedules equally well. 

Results of this fieldtest indicate that frequency response options and time per typical use 
options apply equally well to teachers of traditional and block scheduled classes. During 
earlier field trials of this instrument, we tested two different response categories at the 
frequent end of the response scale. “Almost every day” and “once a session or more often” on 
early trials gave way to “every or almost every period” on later field trials which was 
replaced by the current “more than 1 period per week” used on this instrument. 

Our early drafts lacked suitability for schools using alternative schedules, such as 
block scheduling, in which classes meet for something other than five days per week. 10 At 
that time, some block-scheduled respondents to our early surveys steadfastly defended their 

10 Compared to a traditional schedule, block scheduled classes meet for longer 
periods of time at less frequent intervals. A traditionally-scheduled course might meet daily 
for 50 minutes; a block-scheduled course might meet every other day for 100 minutes. 
Overall contact hours are equal, but classes meet half as often, either two or three times per 
week. 



inability to respond accurately to frequency options that do not relate to their schedule. 
Indeed, results from our early fieldtests suggested that using a conventional scale to 
describe block-scheduled classes confuses the meaning ascribed to response options on the 
high end. "Once or twice a week" could mean every class meeting for block-scheduled 
classes, for example, which makes the meaning of "almost every day," intended to convey 
more often than once or twice a week, even more ambiguous. The revised response options 
apply equally well to both traditional and block schedules and avoid fuzzy interpretation. 
And the loss in distinction between “every day” and “more than once a week” may be no loss 
at all; for example, Burstein et al. (1995) failed to detect significant differences between 
those response options. 

Teachers prefer longer referent periods when responding to questions 
about their teaching practices. Teachers believe that longer referent periods— a 
semester rather than a month or a week— provide better opportunities to indicate the full 
extent of their instructional repertoire than shorter periods. Data from this fieldtest, 
however, indicate that teachers are less accurate when summarizing their practices over a 
semester compared to describing their instructional activities that occurred that day. 

The items fieldtested in this project were developed for possible inclusion in a large 
survey that is mailed out over a period of time, with a potentially heavy concentration 
around December- January. During the project’s instrument design phase, item writers and 
survey designers discussed the pros and cons of appropriate referent periods for the items: 
An accurate reflection of a semester’s worth of instruction will likely yield a more balanced 
portrayal than a shorter one. Because teachers may go through seasonal peaks and valleys 
in the variety of the particular methods they employ, longer referent periods may provide a 
more historically accurate picture of the instructional practices. 

Conversations with the teachers we observed while developing and testing the items 
indicate that teachers prefer longer referent periods, perceiving that they can more 
accurately represent their own teaching style. In discussions peppered with teacher 
disclaimers that the class we just sat through was not really typical because of various 
factors that influence the daily instructional choices teachers make, teachers said they were 
most comfortable responding to questionnaire items that allowed them to indicate the 
extent of the instructional methods and materials they use throughout the semester. 

We chose the referent period for the fieldtest to address two practical concerns. 

First, using “the current semester” would potentially result in referent periods of different 
lengths, since some teachers might respond at the end of a semester while others would 



complete the questionnaire at the beginning of a semester. Alternatively, using a shorter 
referent period, such as “the most recent two weeks,” especially during that time of year 
would likely not be representative of teaching throughout a full semester. The final fieldtest 
version, therefore, asked teachers to respond to items thinking about their instructional 
practices within one particular course during the previous semester. 

Data from items describe mathematics instructional activities generally 
across a semester. A final challenge in adopting these items for large-scale use is to 
determine the magnitude and/or scope of the large-scale data collection in which they would 
be used. Important considerations are the breadth of the survey topics covered, the 
granularity of the data collected, the amount of respondent time available, and the 
characteristics of the intended respondent group. The items tested in this project were 
intended to describe a relatively broad swath of instructional practices used by secondary 
mathematics teachers at a middle-level of detail and to take about 45 minutes of response 
time. For this purpose, the items did not ask for detailed information on each of the 
instructional practices that teachers use within each of several specific cognitive areas (high 
level of detail). Nor were they limited to responses that could be used to create 
generalizations across a few broad areas of mathematics instruction (low level of detail). At 
the level of detail fieldtested, the items as modified are most appropriate for a SASS-type 
questionnaire as a subset of items for a more narrowly defined group of mathematics- 
teaching respondents to collect information that would generally describe course content 
and emphasis, instructional techniques, materials, and classroom context. 

Other large-scale uses are possible, but may require modification from the items 
included in this fieldtest. One potential use might be, for example, to collect more-detailed 
information on the pedagogic practices used to teach a limited number of instructional 
objectives. By more closely targeting potential respondents and maintaining a similar level 
of respondent burden, new appropriate items could be developed from the fieldtested items 
by combining the teacher and/or student instructional practice item(s) with each element of 
the items on lesson content and/or student learning objective. 



Applicability to Other Content Areas 

Developing applicability to instruction in other content areas and across all grade 
levels would potentially provide the means with which to understand better the intricacies 
of all K-12 instruction and clearly extend and improve the usefulness of these items and 
validation methods. This pilot test of instruments and methods to gather data about 



classroom instructional practices was confined to mathematics classes generally available 
for students in eighth to twelfth grades and was not designed specifically to assess the 
extent to which the instrument and methods are appropriate for other disciplines. A review 
of the process and results obtained with eighth to twelfth grade mathematics, however, 
suggests some ideas about applicability of the items and methods to other content areas. 

Considerations about items. Results from this fieldtest suggest no content- 
specific concerns that might indicate this level of data collection should be restricted to 
mathematics. Survey developers with an understanding of the curriculum, learning 
objectives, and practices of other content areas should be able to adapt these of similar items 
to other content areas of interest. 

To a large extent, measuring and obtaining a high degree of reliability and validity of- 
individual instructional practices items depends on the degree to which: (1) the items are 
appropriately specific, narrowly defined, and unambiguous; (2) teachers’ conceptions of their 
practices are sufficiently consistent over time; (3) the terms and language in each item are 
understood in a uniform way by respondents and analysts; and (4) the concept or activity is 
observable by an outsider not directly involved in the classroom activity. 

The instructional practice items that showed consistent results over repeated 
measurements, as determined by comparing teacher responses to the first and second 
questionnaire, suggest first that those items provoked the same defined picture of a specific 
activity and that definition was sustained by the teacher over the period of time between the 
first and second administration of the questionnaire. Items with broader definitions may 
suggest different activities to teachers during repeated readings and may result in different 
responses by the teacher, leading to low reliability. Second, consistent repeated responses 
require that the teachers’ own understanding and thinking about the practices used in their 
classrooms are sufficiently, strong, well-defined, and unwavering between the two 
administrations that the match between the item and their practice remains constant. 
Different'conceptualizations would surely lead to different responses. Third, the 
terminology used to construct each item must convey an identical thought to both survey 
respondents and data analysts. If each group does not understand the item in the same 
way, the resulting interpretation may be incorrect or misleading. Finally, measuring the 
validity of items using classroom observers to understand teacher reports of classroom 
activity depends to a great extent on the degree to which an event is observable by someone 
other than the teacher. As stated earlier, assessing the validity of teachers’ responses to 
items on intended learning objectives for students during the observed class for this study 
was most problematic for objectives that were unobservable unless specifically 



demonstrated. The structure of the teacher/observer relationship was designed such that 
the observer was not told before the observed class the content of the main student learning 
objectives that would be emphasized during that class. To determine the lesson objectives, 
the observer had to rely on observing the teacher and the students and listening to their 
communication. Some teachers were better than others in explicitly telling students the 
day’s learning objectives at the beginning of the class or as the lesson proceeded. As a 
result, the student learning objectives with the lowest levels of agreement between teacher 
and observer were those that were essentially unobservable or that required that the 
observer have knowledge about prior learning experiences (for example whether or not the 
structure of the story problems was unfamiliar to students). 

Considerations about validation methods. Both logs and surveys suffer from 
the inherent biases associated with all self-reporting, but we based our second comparisons 
on the log recordings for three reasons. First, because the logs were completed daily about 
activities occurring within the last eight hours, they are likely to be less susceptible to 
memory faults than retrospectively recording a full semester’s activities at a single sitting. 
Second, if teachers do have a particular shading they wish to cast on any recording of their 
activities, the temptation is likely to be greatest with responses that represent activities 
from a full semester rather than responses that represent activities from a single day. 

Third, our objective was to determine the validity of items on the questionnaire, using log 
data as a primary source of validating information. 

Using classroom log accounts of instructional activities is an accepted method of 
validating the reliability of teacher questionnaire responses, but the results should be 
interpreted cautiously given the organic limitations of the method. In this study, case study 
teachers kept classroom logs for at least four weeks of classes during the middle to end of 
the second semester of the 1996-97 school year. In the best of circumstances, the degree to 
which these log data can reflect the questionnaire data depends on the extent to which that 
four week log period was representative of the semester period covered by the 
questionnaires. For example, if a teacher indicates that she uses three-dimensional 
graphics to demonstrate a concept once or twice per month yet four weeks of daily logs show 
no such activity, we can not ascertain if her questionnaire response was inaccurate or 
whether the case study period was not reflective of her typical teaching over a longer period. 
Directly validating survey responses covering lengthy periods of time is a practical problem 
of not insubstantial proportions. Ideally, of course, one would validate questionnaire 
responses with daily logs over the same time period, but the heavy respondent burden to 
validate data collection throughout an entire semester is a strong deterrent to using that 
method. 



Recommendations 



The recommendations in this section are based on our learning and experiences 
administering this questionnaire to approximately 300 teachers, conducting case studies 
with 41 teachers, and analyzing the resulting data. We use those experiences to make 
recommendations intended to further improve the items and the fieldtest process and to 
identify the items we think warrant continued use to collect information that will enlighten 
our understanding of the design, intention, and implementation of secondary classroom 
instruction in mathematics. The following recommendations are intended to improve the 
survey items and the fieldtest validation process, and to identify items to include in future 
data collections. 

1 . Shorten the referent period for all items to improve respondent 
accuracy, response validation, and ultimately the reliability of the 
responses themselves. This fieldtest collected information from teachers 
about their instructional activities over a semester (questionnaire) , daily over 
four weeks (logs), and following an observed class (daily log). While teachers 
may have different motivations to respond accurately to each type of data 
collection, information obtained in teacher interviews and confirmed by 
fieldtest results indicate that responses across a semester are based on 
general recall while responses to items about daily activities are grounded in 
specific recall. This fieldtest was not designed to produce data to validate 
questionnaire responses with semester-long referent periods, but case study 
teacher interviews indicate what intuition suggests: items requiring a mental 
tally of instructional activities over a semester are based on “best-guess” 
teacher recollection. When asked to recall over a semester, most teachers 
reported they estimated the frequency with which they use instructional 
activities. On the other hand, there was high agreement between teacher and 
observer on both the occurrence and length of time student learning 
objectives, teacher actions, and student activities when responses were 
recorded immediately following the class being described. This may be partly 
the effect of having an outside observer present, but the basic conclusion 
remains: teachers can use immediate recall to respond to questions about 
recent activities, but respond to longer referent items in a more general way. 
Shorter referent periods may be less representative of a single teacher who 
selectively and purposively employs a variety of instructional methods to 
respond to the particular mix of learning objectives throughout a semester- 
long course, but teachers are more likely to accurately recall the events that 
do occur. Compared to validating responses to semester-long referent periods, 
responses to items with shorter referent periods would be cheaper to design 
and conduct and less burdensome to validate, since field trials (and teacher 
burden) would be limited to the length of the referent period. 

2. Redesign the item on course content. Information about the academic 
course content provides the context within which analysts can interpret 



teachers’ reported instructional practices. One major potential use of 
instructional practice data, for example, would be to identify systematic 
variation in instructional methods related to course content. Since previous 
work by Porter and others indicated that course titles provide little indication 
of course content, we attempted to collect specific information on course 
content to complement the data on instructional practices employed. The new 
version of the item we developed for this fieldtest, 1 5 content options over one 
and a half pages, was a compromise between Porter’s very detailed items on 
course content that take multiple pages and our earlier and simpler fieldtest 
item of ten responses on one-half page. 

The responses were disappointing. We had increased the course elements 
from 10 to 15 to provide cleaner distinctions among overlapping areas and 
included descriptions of each element to be even more explicit. While these 
changes may have added clarity to teachers’ thinking as they responded, the 
increase in categories apparently made the necessary addition more complex. 
As reported earlier, ten percent of the responses to the course content 
question were unusable, primarily because they totaled something other than 
100 percent. This suggests that further work is needed to find a format 
suitable for this use. 

Change the format of open-ended items to closed-ended items with 
response options that reflect the range and frequency of expected 
data. Open-ended items in this questionnaire had the worst reliability of all 
other questions for suspected reasons that have already been described. We 
suspect that reliable categorical or ordinal data have more explanatory power 
than noisy continuous data. 

Shorten the number of subitems included in four main questions 
and several others using validation and reliability data generated 
in this fieldtest. Most of the 19 items in this collection contained multiple 
subitems, some of which required two responses per subitem. The item on 
student activities had 24 subitems and required a total of 48 responses. In 
creating and including this item, as with the others, our goal was to collect 
information on a range of practices that would adequately describe the 
diversity of instruction across the country. We assured ourselves that 
respondent information from each subitem would be sufficiently interesting 
and unique as to make it important to collect and analyze. Regardless of the 
potential analytic value in a rich dataset, however, we created items whose 
sheer size may be forbidding to potential respondents. Shortening interesting 
items may require making hard choices, but it may generate more useful 
answers on Items 8 (course content), 13 (teacher activities), 14 (context 
factors), and 15 (student activities). Information from this fieldtest can 
identify subitems with responses that show less reliability than others tested 
and therefore could be dropped. 

Develop items to elicit a finer description of classroom practice 
within specific instructional settings. We designed and fieldtested the 
items reported here to collect information on certain important practices 



across a spectrum of instructional activities and considerations. While they 
paint a reasonably accurate and informative portrait of general mathematics 
instruction within that design, it is also possible to collect considerably finer- 
grained information with greater detail about specific and narrow 
instructional topics. Such a detailed level of data collection was not a goal of 
this fieldtest but is a logical extension that would considerably strengthen our 
understanding about teaching and learning. 

For example, the items reported here collected information on the frequencies 
with which teachers (1) employed each of nine specific student learning 
objectives and (2) included each of 12 different assessment strategies. Data 
collection for a finer description of classroom approach might assess the 
frequency with which teachers employ each assessment strategy for each of 
the specific student learning objectives, asking for 12 responses each on nine 
items. Identifying the intended use of the data is extremely important in this 
respect since such an approach has an obvious and direct impact on 
respondent burden. 

Explore ways to reword items that describe sensitive practices in 
ways that project those activities in an accurate but value-free way 
to reduce response bias from social or professional concerns. Even 
then, treat as suspect responses on certain activities that are obviously 
polemic. Clearly, this is not a new recommendation and these are not new 
concerns. Rather it is a response to information from this fieldtest that 
indicates that some teachers responded to certain questions in ways that 
suggest those answers did not reflect their practice. 

In future whole-SASS fieldtests, use revised instructional practice 
items that can be expected to provide the most information at the 
greatest level of accuracy. Although many of the items provide good 
quality information as currently written, we recommend improving (to greater 
or lesser extents) nearly every item fieldtested. Information from further 
testing can add to our knowledge base about the quality of data the items 
produce. In making these recommendations, we considered information from 
the analyses reported above, especially (1) the degree to which the fieldtest 
information indicated reliable agreements in the concept and length of time 
between the respondent and the observer or daily logs; (2) the frequency with 
which teachers’ indicated they used certain practices or material, in 
conjunction with (3) the extent to which that practice is representative of 
reform or traditional instruction; (4) the clarity of each item’s wording and the 
distinctiveness between like items; and (5) the extent to which certain items 
may show overlap or redundancy in representing certain conceptions of 
teaching as informed by principal components analysis or factor analysis. 

The chart in Appendix M summarizes the recommendations based on our 
overall assessment of how well each meets the above five considerations. 

Design future projects with multiple methods to validate items, 
building on the particular characteristics and strengths of each 
method. Use observation, for example, to validate only those items that 



describe events that can be seen; use artifact analysis to validate items about 
the frequency or content depth of supplementary materials; use interviews to 
understand responses on curricular decisions that cannot be observed. 

9. Design future projects that use identical response options for the 
length of time options on the log sheet and on the questionnaire. 

One practical difficulty with this fieldtest was a critical difference in response 
options between the teacher questionnaire and the case study daily classroom 
log form caused when revisions to the questionnaire were not carried over to 
the log form. The four possible log form response options for the number of 
minutes that teachers (log Item 4) and students (log Item 5) spent on certain 
activities did not match the three possible response time options for the 
parallel Items (13 and 15) on the teacher questionnaire. The validation 
information was not lost, but the relatively straightforward validation task 
became more complex and required analysts to use data on the length of the 
class period to convert log form responses statistically to the metric used on 
the questionnaire item. 

1 0. Design future projects to have a case study period that is consistent 
with the questionnaire referent period. The case studies were too short 
to validate the longest frequency response options on the teacher 
questionnaire. Most teacher questionnaire items asked for teachers to 
respond to questions thinking about their actions within a particular course 
over a full semester. In this fieldtest, case study teachers kept daily logs for 
four weeks, so the extent to which their daily log recordings could validate 
their questionnaire responses depended on the extent to which that particular 
four- week period was representative of the semester. Furthermore, only four 
weeks of daily log data could not be used reliably to validate those teacher 
questionnaire responses that indicated frequencies of “1 or 2 periods per 
semester (or half year)”. This design limitation restricted the extent to which 
we could expect complete validation of any questionnaire responses that 
indicated infrequent use. 

The conscious decision to limit the length of the case studies in this project 
was driven by practical considerations, primarily cost, the degree of burden 
placed on participating teachers, and the practicality of maintaining 
enthusiasm among geographically-dispersed participants over a longer period 
of time. Future fieldtest designs should insist on consistency between 
response options and case study duration. 

Understanding what happens in classrooms is fundamental to improving the 
education of our nation’s students. Using accurate and reliable survey data to measure 
classroom practice can create trust that the resulting national portraits of classroom 
instruction are real and transferable. The results from this fieldtest provide the basis for 
optimism that revised survey items and validation methods can result in the means by 
which we can collect data to describe classroom instructional practices reliably and 
accurately. 
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Classroom Instructional Processes Survey 



This questionnaire asks about the instructional strategies and practices that you use in your math 
classroom. Its purpose is to characterize students’ educational experiences and to inform the 
development of future national surveys of classroom processes. The rationale behind this endeavor 
suggests that in a learning situation the teacher’s actions, the students’ actions, and the instructional 
context-e.g., curriculum, lesson resources-contribute to learning outcomes. This questionnaire 
includes items that ask about these dimensions in a specific course you arc now teaching. 



Our long range study focuses on the instructional content and practices of math courses offered to 
students in grades 8-12, courses which typically cover areas from "prealgebra" through calculus (If 
you do not teach any courses that mainly enroll students in these grades or at those levels please 
return the survey without completing it and add a note explaining why.) When you answer the 
questions, think of a single class for which you had primary responsibility last semester. If you were 
responsible for multiple sections of one or more of the focus courses, select your first such class of 
the day (not homeroom), and think of this as the desi gnated class . The survey begins with questions 
about this d esignated class that provide the context for your answers to later questions about 
instruction. We use the term "period" to indicate a single class meetine. 



If a question is confusing, unclear, or does not seem to fit the activities in your designated class 
please make an explanatory note in the margin on that page. At the conclusion of this questionnaire 
you will be asked to oescribe any sources of confusion. 



The confidentiality of your responses will be protected at every stage of data collection. Formal 

reports and informal conversations about this task will include no reference to information that could 
identify participants or their schools. 



Teaching and Learning Conditions of the School and Designated 
Mathematics Class 

When you answer the questions, think of a single class for which you had primary responsibility last 
semester. If you were responsible for multiple sections of one or more of the focus courses, select your 
first such class of the day (not homeroom), and think of this as the designated^ class . 



What is the course title? 



What is the usual schedule and length (in minutes) of daily class meetings for the d esignated 
class ? If the weekly schedule is normally the same, just complete Week 1. 



Week 1 Week 2 

Monday 


Examples 


Wk 1 Wk 2 

90 


Wk 1 
45 


Tuesday 


90 


45 


Wednesday 


90 


45 


Thursday 


90 


45 


Friday 


90 


45 



In what grade(s) are the students in your designated class enrolled? Enter the number of 
students at each grade level. 



Ungraded 


9th 


6th 


10th 


7th 


11th 


8th 


12th 



Estimate the percentage of students in your designated class who function at each level of 
academic ability in mathematics. Use the best information you have readily available 
(including your own professional judgment). Percentages should total to 100%. 



Much above the SCHOOL average , % 

Somewhat above the SCHOOL average % 

At the SCHOOL average % 

Somewhat below the SCHOOL average % 

Much below the SCHOOL average % 



TOTAL 



100 % 



5. In your opinion, what percentage of the students who enrolled in this class at the beginning 
are capable of learning the required material given available time and resources? 

percent 



6. At this school, how much actual influence do you think teachers have over the following? 





No 

influence 






Complete 

influence 


Setting policy on grouping students in 
classes by ability 


0 1 


2 


3 


4 


Deciding which students take which 
courses 


0 1 


2 


3 


4 


Determining discipline policy 
within your classroom 


0 1 


2 


3 


4 



How much control do you have over each of the following areas of planning and teaching? 
On the left response set, indicate how much control you have through participation or 
representation in department, or school, or district'decision groups; on the right, indicate the 
control you have in your own classroom. 



Through my participation in 

department, school, In my own classroom 

or district decisions 



No 

Contro 




Complete 

Control 




No 

Control 




Complete 

Control 


0 


2 


3 


4 


a. Selecting textbooks 


0 


1 


2 


3 


4 


0 


2 


3 


4 


b. Selecting instructional materials 


0 


1 


2 


3 


4 


0 


2 


3 


4 


c. Selecting content, topics and skills to be 
taught 


0 


1 


2 


3 


4 


0 


2 


3 


4 


d. Selecting teaching techniques 


0 


1 


2 


3 


4 


0 


2 


3 


4 


e. Determining the amount of homework to 
be assigned 


0 


1 


2 


3 


4 


0 


2 


3 


4 


f. Setting standards for grading in your 
classes 


0 


1 


2 


3 


4 


0 1 


2 


3 


4 


g. Disciplining students 


0 


1 


2 


3 


4 
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II. Course Content and Emphasis 



8. Math content areas: Estimate the percentage of time (if any) that you have spent or will 

spend during this course teaching each content area listed below in the designated class . The 
total should equal 100 percent. 

Number Use/Operations 

(Estimation, computational algorithms, fractions, decimals, ratio and 
proportion, percent, real numbers, number theory, order of operations, 

relationships between operations) % 



Measurement 

(Use of measuring instruments, theory [arbitrary, standard, size], 

conversions, metric [SI] system, length, perimeter, area, volume, surface area 

angles, circles [e.g., pi, radius, diameter, area], Pythagorean theorem, mass 

[weight], time, temperature, speed) % 

Consumer Applications 

(Simple interest, compound interest, rates [e.g., discount, commission], 

spread sheets) % 

Data Analysis 

(Bar graphs, histograms, pictographs, line graphs, stem and leaf plots, scatter 

plots, box plots, mean, median, mode, mean deviation, smoothing of graphs) % 

Pre- Algebra 

(Integers, absolute value, exponents, scientific notation, use of variables, 
expressions, evaluation of formulas and expressions, one-step equations, 
coordinate plane) % 

Basic Algebra 

(Multi-step equations, inequalities, literal equations, lines [slope and 

intercept], operations on polynomials, factoring, square roots and radicals, 

operations on radicals, rational expressions) % 

Advanced Algebra 

(Quadratic equations, systems of equations, systems of inequalities, 

compound inequalities, matrices [determinants], conic sections, rational, 

negative exponents, radicals, rules for exponents, complex numbers binomial 

theorem, factor [remainder theorems], field properties of real number system) % 

Functions 

(Notation, relations, linear, quadratic, polynomial, rational, logarithmic, 

exponential, trigonometric/circular, inverse, composition) ' % 

Basic (non-measurement) Geometry 

(Basic terminology, relationships between lines, angles, planes, triangles, 

quadrilaterals, polygons, congruence, similarity, parallels, circles, constructions) % 



Advanced Geometry 

(Logic, reasoning, proof, symmetries, loci, spheres, cones, cylinders, 

polyhedra, 3-dimensional relationships, transformational, coordinate, vectors, 

analytic, non- Euclidean, topology) % 



[Topic list continues on next page] 



Math content areas (cont.): 



Trigonometry 

(Basic ratios, radian measures, right triangle trigonometry, law of Sines, 
Cosines, identities, trigonometric equations, polar coordinates, periodicity, 
amplitude) 



Statistics 

(Variability /standard deviation, quartiles, percentiles, bivariate distributions, 
sampling, confluence intervals, correlation, lines of best fit, hypothesis testing. 
Chi-square, data transformation, central limit theorem) 

Probability 

(Sample spaces, compound probability, conditional probability, independent 
and dependent events, empirical, expected value, binomial distribution, 
normal curve) 

Finite Math/Special Topics 

(Sets, logic, mathematical induction, linear programming, networks, iteration 
and recursion, permutations/combinations, simulations, fractals) 



Analysis 

(Sequences and series, limits, continuity, rates of change, maxima, minima, 
differentiation, integration) 



TOTAL 



9 . 



Dimensions of math: Math courses approach the study of skills and concepts in many ways, and 
material covered during a given period may include several emphases. Listed below are cross- 
cutting dimensions of mathematics that may be covered during any class meeting. First, please 
• circle the response that indicates how frequently activities in the designated class emphasize each 

dimension. Second, circle the response that indicates about how much time you typically spend 
on that dimension when you do include it. (For example, if you usually open class meetings with 
a short, real-life application problem, you would circle 4 for "more than once a week" and a for 
"10 minutes or less.”) Do not answer the "time" item if your "frequency" response is "never." 

^ If your school is not on semesters, answer for half a year. 



ft 



i 



I 



ft 



I 



ft 



Lesson 


i content emphasis: 


Never 


Frequency of Use 

I or 2 
periods 

per 1 or 2 One 

semester periods period 

(or half per per 

year) -month week 


> 1 
period 
per 
week 


Time Per Typical Use 

<.10 < '/z >_ Vi 

minutes period period 


a. 


Computational techniques 


0 


i 


2 


3 


4 


a 


b 


c 


b. 


Specific math facts or 
definitions 


0 


i 


2 


3 


4 


a 


b 


c 


c. 


Broad math concepts 


0 


i 


2 


3 


4 


a 


b 


c 


d. 


Problem solving/inquiry 
skills 


0 


i 


2 


3 


4 


a 


b 


c 


e. 


Skill in communicating in 
speech or writing about 
mathematical ideas or 
applications 


0 


i 


2 


3 


4 


a 


b 


c 


f. 


Importance of 
mathematics in daily life 


0 


i 


2 


3 


4 


a 


b 


c 


g- 


Applications of 
mathematics in science 


0 


i 


2 


3 


4 


a 


b 


c 


h. 


Applications of 
mathematics in business 
and industry 


0 


i 


2 


3 


4 


a 


b 


c 



ft 
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62 



10. Objectives; During the year, iessons in a course vary in the difficulty and complexity of their 
objectives for student learning, and in the strategies used to achieve them. Listed below are 
typical learning objectives. Circle the response that indicates for each objective the average 
frequency with which it occurs in the designated class . 



Student learning objective: 

a. Memorize (e.g., facts, definitions, or formulae) 

b. Understand (e.g., concepts, relationships, 
theorems) 

c. Collect data (e.g., observe, measure, count) 

d. Order, compare, estimate, approximate 

e. Perform mathematical operations, execute 
algorithms, classify 

f. Solve "story" problems with familiar 
structures, replicate proofs 

g. Recognize and solve story problems 
with unfamiliar or complex structures 

h. Interpret data (e.g., charts, graphs, tables) 

i. Build and revise theories, develop proofs 



Never 


i or j. 
periods 
per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 

period 

per 

week 


0 


1 


2 


3 


4 


0 


1 


2 


3 


4 


0 


1 


2 


3 


4 


0 


1 


2 


3 


4 


0 


1 


2 


3 


4 


0 


1 


2 


3 


4 


0 


1 


2 


3 


4 


0 


1 


2 


3 


4 


0 


1 


2 


3 


4 



11. Assessment content: On the tests, quizzes, projects, portfolio products, and other formal 

assessment strategies you use in this class, how often do the following kinds of items appear? 



a. Items that require students to compute answers 

b. Items that require students to recognize or recall 
definitions or concepts 

c. Items that allow more than one solution or 
approach 

d. Items that require more than one step to reach a 
solution 

e. Items that require students to explain concepts in 
words or pictures 

f. Items that require students to observe or measure 

g. Items that require students to compare an 
estimated answer with a calculated or computed 
answer 

h. Items that require the use of algorithms to solve 
problems 

i. Items that require students to describe how they 

solved problems by showing all their work, 
writing explanations, or drawing pictures 

j. Items that require students to interpret tabular or 
graphical data 

k. Items that require the application of familiar 
concepts or principles to different or unfamiliar 
situations 

l. Items that require a critique or analysis of a 

suggested solution to a problem 

m. Other (specify) 



Never 


Only during 
units 

focusing o:i 
that skill 


Occasionally 


Average of 
once per 
assess mail 


Twice or 
more per 
assessment 


0 


l 


2 


3 


4 


0 


i 


2 


3 


4 


0 


i 


2 


3 


4 


0 


i 


2 


3 


4 


0 


l 


2 


3 


4 


0 


l 


2 


3 


4 


0 


l 


2 


3 


4 


0 


l 


2 


3 


4 


0 


l 


2 


3 


4 


0 


l 


2 


3 


4 


0 


l 


2 


3 


4 


0 


l 


2 


. 3 


4 


0 


l 


2 


3 


•' 4 
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Indicate the extent to which this course~as it is taught to the single designated class — is 
interdisciplinary or integrated with another course, if at all. By this we mean that certain topics are 
taught together to enhance the depth of both, such as combining math and science units that rely on 
statistical analysis or using mathematical skills to represent and understand historical or political 
trends. Circle the appropriate response on each line. 



a. I use examples or illustrations from other 
subject areas in my math teaching, e.g., 
in "story" or other application problems. 





1 or 2 


1 or 2 


One 


> 1 




periods 


periods 


period 


period 




per 


per 


per 


per 


Never 


semester 


month 


week 


week 


0 


l 


2 


3 


4 



b. 1 blend the teaching of another subject 0 

area into my math teaching in this class. 



2 3 4 



c. I work with teachers from one or more 0 12 3 

subject areas to integrate other subjects 
into my math teaching. 



d. A subject area I use more often than others to illustrate math applications is 



III. Instruction 
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Teacher activity: Different instructional strategies involve different types of teacher activity during a 
lesson. For each teacher activity listed below, first circle the response that indicates how often you 
do it with your designated class , and then circle the letter that indicates about how much total time 
per period you typically spend when you do it. (You may engage in more than one activity during a 
class session.) Do not answer the "time" item if your "frequency" response is "never." 



Frequency of Use 



a. Lecture, perhaps occasionally soliciting 


Never 


1 or 2 

periods 

per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per 
week 


<. 10 
minutes 


< Vi 
period 


'/2 

period 


brief student input or using the board or 
overhead projector to highlight a key 
term or present an outline 


0 


1 


2 


3 


4 


a 


b 


c 


b. Demonstrate a concept, using two- 
dimensional graphics such as drawings 
on the board, overhead projector, or 
computer 


0 


1 


2 


3 


4 


a 


b 


c 


c. Demonstrate a concept, using three- 
dimensional tools such as 
manipulatives, models, or other objects 


0 


1 


2 


3 


4 


a 


b 


c 


d. Lead students in recitation, drills, or 
question-and-answer sessions 


0 


1 


2 


3 


4 


a 


b 


c 


e. Observe or monitor student-led whole 
class discussions or demonstrations 
related to mathematics 


0 


1 


2 


3 


4 


a 


b 


c 


f. Provide individual or small group 
tutoring as needed during individual 
seatwork or small group activities 
involving everyone in class 


0 


1 


2 


3 


4 


a 


b 


c 


g. Provide remedial or enriching 

instruction to a pull-out group while the 
rest of the class works on assignments 


0 


I 


2 


3 


4 


a 


b 


c 


h. Work on administrative tasks, such as 
recordkeeping, while students work on 
assignments 


0 


I 


2 


3 


4 


a 


b 


c 


i. Administer a test or a quiz 


0 


1 


2 


3 


4 


a 


b 


c 


j. Stimulate student discussions of 
approaches to solving problems, 
explanations of their mathematical 
thinking, or open-ended questions 


0 


1 


2 


3 


4 


a 


b 


c 


k. Demonstrate uses of technology in 
mathematics 


0 


I 


2 


3 


4 


a 


b 


c 


1. Set up and monitor or supervise 
cooperative learning activities 


0 


I 


2, 


3 


4 


a 


b 


c 
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Context factors: Listed below are two kinds of factors that may characterize a class and, where 
present, may require accommodation in instruction. The first set of factors are student characteristics 
that contribute to diversity. The second set are usually viewed as problematic. Both types vary quite 

a lot across schools, districts, and regions. For each factor, please indicate whether it is present to 
any degree. 



No 

Student Diversity 

a. Students have very different 

academic abilities in this subject. 0 

b. Students have different backgrounds, 0 

e.g., economic, language, 
culture. 

c. Some students have special needs, 0 

e.g., physical or emotional 
impairment. 

Problems 

d. Students are inadequately prepared 



for level of material. 0 

c. Students are disinterested. 0 

f. There arc threats to students’ 

personal safety. o 

g. There is a shortage of appropriate 

technology for student use. 0 

h. There is a shortage of other 

equipment for student use. 0 

i. Students have low morale. 0 

j. Parents are not interested in their 0 

child’s learning. 

k. Family problems preoccupy students. 0 

l. There is a shortage of appropriate 

technology for teacher use. 0 

m. There is a shortage of other 

equipment for teacher use. 0 

n. There is a shortage of preparation 

and planning time for teachers. 0 



Present in Class? 

Minimally Moderately Considerably 



1 2 3 

1 2 3 

1 2 3 



1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 
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Context factors (cont.): 



Present in Class? 

No Minimally Moderately Considerably 

Problems (cont.) 

o. There are threats to teachers’ 

personal safety. 0 1 2 3 

p. Physical facilities are inadequate. 0 1 2 3 

q. Administrative support for instruction 

is inadequate. 0 1 2 3 
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Student activities: Listed beiow are many kinds of student lesson activities, some with a long history 
of use in math classes and others associated with recent reform efforts. First, scan the whole list 
from a to x to see the array of choices, then circle the number next to each that indicates how often— 
if at all— the way you teach the designated class during the regular class period requires that students 
engage in each activity. Do not answer the "time" item if your "frequency" response is "never." 







Frequency of Use 




Time 1 


Per Typical Use 




Never 


1 or 2 

periods 

per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per 
week 


<. 10 
minutes 


< Vi 

period 


A Vi 
period 


a. Listen to or observe 
teacher presentations 


0 


i 


2 


3 


4 


a 


b 


c 


b. Work individually on 
exercises, worksheets, or 
workbooks 


0 


i 


2 


3 


4 


a 


b 


c 


c. Use a textbook for 
activities other than 
homework, e.g., reading 
or reference 


0 


i 


2 


3 


4 


a 


b 


c 


d. Use supplementary printed 
materials 


0 


i 


2 


3 


4 


a 


b 


c 


e. Participate in whole-class 
discussions 


0 


i 


2 


3 


4 


a 


b 


c 


f. Explain how what they 
learn in class relates to the 
real world 


0 


i 


2 


3 


4 


a 


b 


c 


g. Engage in on-task 

discussion primarily with 
other students 


0 


i 


2 


3 


4 


a 

> 


b 


c 


h. Correct or review 

previous day’s homework 


0 


i 


2 


3 


4 


a 


b 


c 


i. Work on 

projects/assignments that 
take a week or more to 
finish 


0 


i 


2 


3 


4 


a 


b 


c 


j. Give or listen to other 
students give oral reports 


0 


i 


2 


3 


4 


a 


b 


c 


k. Solve problems for which 
there are several 
appropriate answers or 
approaches 


0 


i 


2 


3 


4 


a 


b 


c 


1. Write a report or paper 


0 


i 


2 


3 


4 


a 


b 


c 
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15. Student activities (cont.):... Indicate how often— if at all— the lesson activities you use with the designated 
class during the regular class period require that students : 



Frequency of Use 





Never 


1 or 2 

periods 

per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per 
week 


m. Work in small groups 


0 


1 


2 


3 


4 


n. Do lab or field work 


0 


1 


2 


3 


4 


o. Evaluate other students’ work 


0 


1 


2 


3 


4 


p. Practice or drill on 
computational skills 


0 


1 


2 


3 


4 


q. Work on assignments that will 
be due as homework on the 
next day 


0 


1 


2 


3 


4 


r. Respond orally to questions 
testing recall 


0 


1 


2 


3 


4 


s. Respond orally to open-ended 
questions 


0 


1 


2 


3 


4 


t. Take notes 


0 


1 


2 


3 


4 


u. Explain to the whole class 
solutions developed 
individually or in small 
groups 


0 


1 


2 


3 


4 


v. Participate in structured 
cooperative learning activities 


0 


1 


2 


3 


4 


w. Take tests, quizzes, other 
assessments 


0 


1 


2 


3 


4 


x. Wait for completion of non- 


0 


1 


2 


3 


4 



academic tasks, e.g., 
attendance, homework 
collection, behavior 
management, etc. 



Time Per Typical Use 



<.10 < 'h _> 'h 

minutes period period 



a 

a 



b 

b 

b 

b 



b 

b 



b 

b 

b 



c 

c 

c 

c 



c 

c 
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Non-academic time: How many minutes during each meeting of your designated class do you 
typically spend on activities not directly related to the academic content of this course? (Examples: 
listening to or making announcements, taking attendance, establishing rapport, or handling 
disruptions.) 

minutes 



Demonstrating math competencies: As a result of successful study, students are able to demonstrate * 
proficiency in tasks of different levels of difficulty and complexity. In the designated class , how 
often do students have formal or informal opportunities to demonstrate the following competencies 1 
Circle one response on each line. 

4 





Never 


1 or 2 
periods 
per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

perioc 

per 

week 


> i 
pen<>: 
per -re 


a. Memorize basic number facts 
or formulae 


0 


l 


2 


3 


4 


b. Understand a math concept 


0 


l 


2 


3 


4 


c. Collect original data 


0 


l 


2 


3 


4 


d. Organize, summarize, or display 
information 


0 


1 


2 


3 


4 


e. Use estimation to determine the 
reasonableness of results 


0 


l 


2 


3 


4 

s 


f. Perform computations using basic 
number facts 


0 


l 


2 


3 


4 


g. Use algebraic or geometric reasoning 
strategies to solve problems 


0 


l 


2 


3 


4 


h. Analyze and interpret information 


0 


l 


2 


3 


4 


i. Generalize from patterns or examples 


0 


l 


2 


3 


4 


j. Generate original examples 


0 


l 


2 


3 


4 



t 
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IV. Materials 



Teacher materials: Indicate the extent to which the materials and equipment listed below are 
available, and the frequency with which you use them to teach your designated class . Indicate 
frequency of use only if the item was available. (Note, a separate item will ask how often students 
use them.) 

0 = The item was not available 

1 = Item was available in limited supply or with limited functions so that use was less 

effective than it could have been. 

2 = Item was available in adequate supply and/or with appropriate functions for 

most effective use. 



Availability Frequency of Use 





Not 

available 


Limited 

availability 


Full 

availability 


Never 


1 or 2 

periods 

per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per week 


a. Board 


0 


1 


2 


0 


1 


2 


3 


4 


b. Graph paper 


0 


1 


2 


0 


1 


2 


3 


4 


c. Protractors, rulers, or compasses 


0 


1 


2 


0 


1 


2 


3 


4 


d. Appropriate calculator 


0 


1 


2 


0 


1 


2 


3 


4 


e. Overhead projector 


0 


1 


2 


0 


1 


2 


3 


4 


f. Film or videotape 


0 


1 


2 


0 


1 


2 


3 


4 


g. Computer or computer programs 


0 


1 


2 


0 


1 


2 


3 


4 


h. Manipulatives, models, or other 
objects 


0 


1 


2 


0 


1 


2 


3 


4 


i. Other (specify) 


0 


1 


2 


0 


1 


2 


3 


4 



» 




BEST COPY AVAILABLE 
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Student materials: Indicate the extent to which materials and equipment listed below are 
available, and the frequency with which students in your designated class use them during a 
regular class period. Indicate frequency of use only if the item was available. 

0 = The item was riot available • 

1 — Item was available in limited supply or with limited functions so that use was less 

effective than it could have been. 

2 = Item was available in adequate supply and/or with appropriate functions for 

most effective use. 

4 









Availability 




Frequency of Use 






Not 

Available 


Limited 

Availability 


Full 

Availability 


Never 


l or 2 
periods 
per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> l 
period 
per 
week 


a. 


Board 


0 


I 


2 


0 


1 


2 


3 


4 


b. 


Graph paper 


0 


1 


2 


0 


1 


2 


3 


4 


c. 


Protractors, rulers, or 
compasses 


0 


1 


2 


0 


1 


2 


3 


4 


d. 


Appropriate calculator 


0 


1 


2 


0 


1 


2 


3 


4 


e. 


Overhead projector 


0 


1 


2 


0 


1 


2 


3 


4 


f. 


Film or videotape 


0 


1 


2 


0 


1 


2 


3 


4 


£ 


Computer or computer 
programs 


0 


1 


2 


0 


1 


2 


3 


4 


h. 


Manipulatives, models, or 
other objects 


0 


1 


2 


0 


1 


2 


3 


4 


i. 


Other (specify) 


0 


1 


2 


0 


1 


2 


3 


4 



4 



« 



4 



4 



4 



i 



4 




i ; 
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The following items concern the questionnaire itself. Information you provide will help us improve 
the questionnaire for future use. 

20. Were any of the questions confusing or unclear? 

No 

Yes If yes, please list the question number and describe the source of confusion. 

Number Source of confusion 



21. Use the space below to describe any other problems or make any recommendations about the 
questionnaire. 



Please remember to note the time you finish and record the total time you spent on this survey on the 
front page. Thank you again for your help. 



Appendix B 



Fieldtest Sample and Response 



Appendix B 

Mail Survey Fieldtest Sample 





Percent 


Percent 




of all 


of all 


Number 


surveys 


in scope 




mailed 


surveys 



Total surveys mailed 


Public school teachers 


264 


63 


Private school 


154 


37 


teachers 


Total 


418 


100 


Out of scope 


Does not teach math 


18 




On leave 


3 




Total 


21 


5 



Total in scope 


397 


95 


100 


Refused 


13 


3 


3 


Completed and returned 


Public school teachers 


191 


46 


48 


Private school 


105 


25 


26 


teachers 


Total 


296 


71 


75 


Not returned 


88 


21 


22 



Total Schools 


Invited 


Participated 


Public Schools 


87 ' 


76 


Private Schools 


77 


64 




B-l 
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Instructional Processes Daily Classroom Log 
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Appendix D 

Mail Questionnaire Item Response Rates* 



Item Number 



Item 

Response 

Rate 



Item 

Use 

Rate 



1 

2 

3 

4 

5 

6 



(a) 


(b) 

97.3 
100.0 

88.2 

97.0 

99.3 


(c) 


a 


99.0 




b 


98.6 




c 


98.6 




al 


95.3 




bl 


94.3 




cl 


93.9 




dl 


94.3 




el 


93.2 




fl 


94.6 




gl 


94.6 




a2 


97.6 




b2 


98.3 




c2 


99.3 




d2 


99.7 




e2 


99.3 




f2 


99.3 




g2 


99.3 

89.5 




a 




81.8 


b 




81.1 


c 




58.8 


d 




66.9 


e 




82.1 


f 




86.5 


g 




61.5 


h 




57.1 


i 




65.5 


j 




38.2 


k 




46.6 


1 




27.0 


m 




42.6 


n 




28.0 


0 




27.0 


al 


97.0 


95.3 


bl 


97.3 


96.3 


cl 


95.9 


94.9 


dl 


98.3 


98.3 


el 


97.0 


86.5 


fl 


98.0 


97.0 



* See Appendix K for formulas used in calculations. 
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Appendix D 

Mail Questionnaire Item Response Rates* 





Item 


Item 


Item Number 


Response 


Use 




Rate 


Rate 


(a) 


(b) 


(C) 


gl 


97.6 


93.2 


• hi 


97.6 


92.6 


9 a2 


98.9 




b2 


99.3 




c2 


99.3 




d2 


98.6 




e2 


99.6 




f2 


99.3 




g2 


100.0 




h2 


99.3 




10 a 


99.3 


93.9 


b 


98.6 


98.6 


c 


99.3 


88.2 


d 


97.3 


93.9 


e 


99.3 


98.6 


f 


99.7 


98.0 


g 


99.0 


93.2 


h 


99.0 


93.6 


i 


98.6 


69.6 


11 a 


100.0 


100.0 


b 


99.7 


99.7 


c 


99.3 


97.6 


d 


99.7 


99.7 


e 


99.3 


91.9 


f 


99.7 


87.5 


g 


99.3 


89.9 


h 


98.6 


90.9 


i 


99.7 


98.0 


j 


99.3 


90.2 


k 


99.3 


95.3 


1 


99.0 


70.9 


12 a 


99.0 


94.9 


b 


100.0 


80.1 


c 


99.7 


49.7 


13 al 


99.3 


96.6 


bl 


99.3 


98.6 


cl 


99.3 


91.9 


dl 


99.0 


86.1 


el 


98.6 


73.0 


fl 


99.7 


97.0 


gl 


98.6 


67.9 


hi 


98.6 


56.8 


il 


99.0 


99.0 


jl 


97.6 


96.3 



* See Appendix K for formulas used in calculations. 





Appendix D 



Mail Questionnaire Item Response Rates* 





Item 


Item 


Item Number 


Response 


Use 




Rate 


Rate 


(a) 


(b) 


(C) 


kl 


99.3 


91.6 


11 


99.3 


86.8 


13 a2 


98.6 




b2 


98.3 




c2 


98.2 




d2 


98.0 




e 2 


99.5 




f2 


98.3 




g2 


99.0 




h2 


97.6 




i2 


95.9 




j2 


97.2 




kl 


98.5 




12 


97.7 




14 a 


99.7 




b 


100.0 




c 


100.0 




d 


99.7 




e 


99.7 




f 


100.0 




g 


99.7 




h 


99.3 




i 


99.7 




j 


100.0 




k 


100.0 




1 


99.7 




m 


99.7 




n 


99.3 




0 


100.0 




P 


99.7 




q 


99.7 




15 al 


100.0 


97.0 


bl 


99.7 


95.9 


cl 


98.6 


79.4 


dl 


99.3 


94.9 


el 


99.7 


97.6 


fl 


99.3 


91.6 


gl 


99.3 


93.6 


hi 


100.0 


98.0 


il 


99.0 


57.4 


jl 


99.7 


41.2 


kl 


99.3 


95.6 


11 


99.7 


39.2 


ml 


99.7 


96.3 



* See Appendix K for formulas used in calculations. 
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Mail Questionnaire Item Response Rates* 





Item 


Item 


Number 


Response 


Use 




Rate 


Rate 


(a) 


(b) 


(C) 


nl 


98.3 


42.2 


ol 


99.0 


60.8 


pi 


97.6 


81.1 


qi 


99.3 


92.6 


rl 


99.0 


95.9 


si 


97.6 


91.2 


tl 


99.3 


95.6 


ul 


99.3 


86.8 


vl 


98.0 


81.4 


wl 


99.0 


99.0 


xl 


97.3 


77.4 


a2 


98.3 




b2 


98.9 




c2 


98.3 




d2 


98.6 




e2 


99.3 




f2 


99.3 




g2 


98.9 




h2 


97.6 




i2 


97.7 




j2 


98.4 




k2 


97.9 




12 


99.1 




m2 


97.5 




n2 


98.4 




o2 


98.3 




p2 


97.1 




q2 


98.5 




r2 


98.2 




s2 


98.5 




t2 


98.6 




u2 


97.7 




v2 


97.5 




w2 


96.6 




x2 


99.6 






99.0 




a 


98.6 


91.9 


b 


99.3 


99.3 


c 


98.0 


76.7 


d 


99.0 


91.9 


e 


99.3 


95.6 


f 


99.3 


97.0 


g 


99.3 


99.0 


h 


99.3 


98.3 



* See Appendix K for formulas used in calculations. 
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Mail Questionnaire Item Response Rates* 



Item Number 



Item 

Response 

Rate 



Item 

Use 

Rate 



§ 



» 



» 



(a) 


(b) 


(c) 


i 


99.0 


96.3 


j 


99.0 


88.5 


al 


99.7 




bl 


98.0 




cl 


97.6 




dl 


98.3 




el 


99.3 




fl 


97.3 




gl 


98.6 




hi 


99.0 




a2 


100.0 


98.3 


b2 


99.3 


88.2 


c2 


98.6 


77.7 


d2 


99.6 


87.8 


e2 


99.3 


76.7 


f2 


99.2 


35.5 


g2 


98.3 


39.5 


h2 


99.6 


70.9 


al 


98.6 




bl 


98.6 




cl 


97.6 




dl 


98.3 




el 


97.6 




fl 


95.6 




gl 


97.3 




hi 


98.3 




a2 


99.3 


92.2 


b2 


97.9 


87.8 


c2 


98.6 


77.0 


d2 


98.9 


87.8 


e2 


98.0 


43.6 


f2 


97.5 


4.4 


g2 


98.6 


27.7 


h2 


98.8 


56.4 



» 




* See Appendix K for formulas used in calculations. 
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Mail Survey Responses (N=296)' 

I. Teaching and Learning Conditions of the School and Designated 
Mathematics Class 

When you answer the questions, think of a single class for which you had primary responsibility last 
semester . If you were responsible for multiple sections of one or more of the focus courses, select your 
first such class of the day (not homeroom), and think of this as the designate d class . 

1 . What is the course title? 



2. What is the usual schedule and length (in minutes) of daily class meetings for the designated 
class ? If the weekly schedule is normally the same, just complete Week 1. 



Week 1 Week 2 

Monday 


Examples 


Wk 1 Wk 2 ! Wk 1 

1 

90 i 45 


Tuesday 


1 

90 ! 45 


Wednesday 


h 

90 i 45 


Thursday 


1 

90 i 45 


Friday 


1 

90 1 45 



3. In what grade(s) are the students in your desjgnateddass enrolled? Enter the number of 
students at each grade level. 



Ungraded 


9th 


6th 


10 th 


7th 


11th 


8th 


12 th 



Estimate the percentage of students in your designated class who function at each level of 
academic ability in mathematics. Use the best information you have readily available 
(including your own professional judgment). Percentages should total to 100%. 



Much above the SCHOOL average 26. 1 

Somewhat above the SCHOOL average 22.6 

At the SCHOOL average 30.6 

Somewhat below the SCHOOL average 13 . l 

Much below the SCHOOL average 7.4 

TOTAL 100.0 



1 See Appendix D for mail questionnaire item response rates. 



5. In your opinion, what percentage of the students who enrolled in this class at the beginning are 
capable of learning the required material given available time and resources? 

x = 89.8 percent 



6. At this school, how much actual influence do you think teachers have over the following? 





No 








Complete 




influence 








influence 


a. Setting policy on grouping students in 


14.6 


16.3 


21.4 


31.0 


23.8 


classes by ability 












b. Deciding which students take which 


6.8 


12.2 


28.9 


40.5 


10.9 


courses 












c. Determining discipline policy 


1.0 


3.1 


8.9 


36.9 


49.8 



within your classroom 



7. How much control do you have over each of the following areas of planning and teaching? On 
the left response set, indicate how much control you have through participation or 
representation in department, or school, or district decision groups; on the right, indicate the 
control you have in your own classroom. 



Through my participation in 

department, school, In my own classroom 

or district decisions 



No 

Control 




Complete 

Control 




No 

Control 






Complete 

Control 


10.2 


8.1 


22.2 


40.5 


18.3 


a. Selecting textbooks 


13.8 


8.3 


16.2 


37.6 


23.8 


5.7 


7.4 


20.2 


42.9 


22.7 


b. Selecting instructional 
materials 


0.3 


2.1 


9.6 


32.5 


55.1 


9.3 


10.4 


23.9 


38.9 


16.8 


c. Selecting content, topics and 
skills to be taught 


6.4 


4.1 


18.0 


37.6 


33.6 


8.6 


6.8 


17.1 


30.0 


37.1 


d. Selecting teaching techniques 


0.0 


0.3 


2.7 


16.9 


80.0 


10.5 


6.1 


18.4 


27.8 


36.8 


e. Determining the amount of 
homework to be assigned 


0.3 


0.7 


2.4 


18.3 


78.0 


8.9 


8.2 


22.3 


32.6 


27.3 


f. Setting standards for grading 
in your classes 


0.3 


1.0 


7.5 


27.5 


63.4 


9.2 


11.0 


25.9 


33.0 


20.2 


g. Disciplining students 


0.0 


1.0 


8.8 


42.0 


47.8 



0 




E-2 



30 



II. Course Content and Emphasis 



Math content areas: Estimate the percentage of time (if any) that you have spent or will spend 
during this course teaching each content area listed below in the designated class The total 
should equal 100 percent. 



Number Use/Operations 

(Estimation, computational algorithms, fractions, decimals, ratio and 
proportion, percent, real numbers, number theory, order of operations, 
relationships between operations) 

Measurement 

(Use of measuring instruments, theory [arbitrary, standard, size], 
conversions, metric [SI] system, length, perimeter, area, volume, surface area 
angles, circles [e.g., pi, radius, diameter, area], Pythagorean theorem, mass 
[weightj, time, temperature, speed) 

Consumer Applications 

(Simple interest, compound interest, rates [e.g., discount, commission], 
spread sheets) 

Data Analysis 

(Bar graphs, histograms, pictographs, line graphs, stem and leaf plots, scatter 
plots, box plots, mean, median, mode, mean deviation, smoothing of graphs) 

Pre-Algebra 

(Integers, absolute value, exponents, scientific notation, use of variables, 
expressions, evaluation ot formulas and expressions, one-step equations, 
coordinate plane) 

Basic Algebra 

(Multi-step equations, inequalities, literal equations, lines [slope and 
intercept], operations on polynomials, factoring, square roots and radicals, 
operations on radicals, rational expressions) 

Advanced Algebra 

(Quadratic equations, systems of equations, systems of inequalities, 
compound inequalities, matrices [determinants], conic sections, rational, 
negative exponents, radicals, rules for exponents, complex numbers binomial 
theorem, factor [remainder theorems], field properties of real number system) 

Functions 

(Notation, relations, linear, quadratic, polynomial, rational, logarithmic, 
exponential, trigonometric/circular, inverse, composition) 

Basic (non-measurement) Geometry 

(Basic terminology, relationships between lines, angles, planes, triangles, 
quadrilaterals, polygons, congruence, similarity, parallels, circles, constructions) 

Advanced Geometry 

(Logic, reasoning, proof, symmetries, loci, spheres, cones, cylinders, 
polyhedra, 3-dimensional relationships, transformational, coordinate, vectors, 
analytic, non-Euclidean, topology) 



% 



% 



% 



% 



% 



% 



[Topic list continues on next page] 



Math content areas (cont.): 



Trigonometry 

(Basic ratios, radian measures, right triangle trigonometry, law of Sines, 
Cosines, identities, trigonometric equations, polar coordinates, periodicity, 
amplitude) 



Statistics 

(Variability /standard deviation, quartiles, percentiles, bivariate distributions, 
sampling, confluence intervals, correlation, lines of best fit, hypothesis testing. 
Chi-square, data transformation, central limit theorem) 

Probability 

(Sample spaces, compound probability, conditional probability, independent 
and dependent events, empirical, expected value, binomial distribution, 
normal curve) 

Finite Math/Snecial Topics 

(Sets, logic, mathematical induction, linear programming, networks, iteration 
and recursion, permutations/combinations, simulations, fractals) 



Analysis 

(Sequences and series, limits, continuity, rates of change, maxima, minima, 
differentiation, integration) 



TOTAL 



9 . 



Dimensions of math: Math courses approach the study of skills and concepts in many ways, and 
material covered during a given period may include several emphases. Listed below are cross- 
cutting dimensions of mathematics that may be covered during any class meeting. First, please 
circle the response that indicates how frequently activities in the designated class emphasize each 
dimension. Second, circle the response that indicates about how much time you typically spend on 
that dimension when you do include it. (For example, if you usually open class meetings with a 
short, real-life application problem, you would circle 4 for "more than once a week" and a for ”10 
minutes or less. ) Do not answer the "time" item if your "frequency" response is "never.” If your 
school is not on semesters, answer for half a year. 



Frequency of Use Time Per Typical Use 

1 or 2 
periods 







Never 


per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per 
week 


<. 10 
minutes 


< 'h 

period 


>L */2 

period 


Lesson 


content emphasis: 


















a. 


Computational techniques 


1.7 


6.3 


9.8 


17.1 


65.2 


38.7 


32.3 


29.0 


b. 


Specific math facts or 
definitions 


1.0 


3.8 


12.1 


27.7 


55.0 


48.4 


35.1 


16.1 


c. 


Broad math concepts 


1.1 


4.6 


21.1 


33.3 


39.6 


35.2 


39.5 


24.9 


d. 


Problem solving/inquiry 
skills 


0.0 


1.4 


11.7 


33.3 


53.6 


17.4 


39.6 


42.7 


e. 


Skill in communicating in 
speech or writing about 
mathematical ideas or 
applications 


10.8 


23.3 


30.0 


19.9 


16.0 


47.5 


37.8 


14.7 


f. 


Importance of 
mathematics in daily life 


1.0 


13.8 


32.1 


29.0 


24.1 


64.3 


26.9 


8.4 


g- 


Applications of 
mathematics in science 


4.5 


16.3 


40.1 


29.4 


9.7 


61.6 


26.8 


11.6 


h. 


Applications of 


5.2 


21.5 


36.3 


24.9 


12.1 


56.7 


30.9 


12.4 



mathematics in business 
and industry 



10 . 



Objectives: During the year, lessons in a course vary in the difficulty and complexity of their 
objectives for student learning, and in the strategies used to achieve them. Listed below are typical 
learning objectives. Circle the response that indicates for each objective the average frequency 
with which it occurs in the designated class . 



Student learning objective: 

a. Memorize (e.g., facts, definitions, or formulae) 

b. Understand (e.g., concepts, relationships, 
theorems) 

c. Collect data (e.g., observe, measure, count) 

d. Order, compare, estimate, approximate 

e. Perform mathematical operations, execute 
algorithms, classify 

f. Solve "story" problems with familiar 
structures, replicate proofs 

g. Recognize and solve story problems 
with unfamiliar or complex structures 

h. Interpret data (e.g., charts, graphs, tables) 

i. Build and revise theories, develop proofs 



Never 


1 or 2 
periods 
per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per 
week 


5.4 


15.6 


24.8 


28.9 


25.2 


0.0 


1.0 


3.8 


14.0 


81.2 


11.2 


25.5 


39.1 


19.0 


5.1 


3.5 


10.8 


34.4 


31.6 


19.8 


0.7 


2.7 


9.2 


22.8 


64.6 


1.7 


7.1 


23.1 


37.6 


30.5 


5.8 


13.7 


35.8 


29.7 


15.0 


5.5 


18.1 


41.6 


24.6 


10.2 


29.4 


25.6 


21.8 


15.4 


7.5 
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Assessment content: On the tests, quizzes, projects, portfolio products, and other formal 
assessment strategies you use in this class, how often do the following kinds of items appear? 



Only during 





Never 


units 

focusing on 
that skill 


Occasionally 


Average of 
once per 
assessment 


T wice or 
more per 
assessment 


a. Items that require students to compute answers 


0.0 


1.7 


3.0 


7.1 


88.2 


b. Items that require students to recognize or recall 
definitions or concepts 


0.0 


3.1 


13.6 


23.1 


60.3 


c . Items that allow more than one solution or 
approach 


1.7 


5.1 


51.7 


31.9 


39.3 


d. Items that require more than one step to reach a 
solution 


0.0 


2.4 


5.1 


16.3 


76.3 


e. Items that require students to explain concepts in 
words or pictures 


7.5 


14.3 


39.1 


23.5 


15.6 


f. Items that require students to observe or measure 


12.2 


28.8 


36.3 


17.6 


5.1 


g. Items that require students to compare an 
estimated answer with a calculated or computed 
answer 


9.5 


21.1 


38.1 


20.4 


10.9 


h. Items that require the use of algorithms to solve 
problems 


7.9 


8.9 


22.9 


21.6 


33.7 


i. Items that require students to describe how they 
solved problems by showing all their work, 
writing explanations, or drawing pictures 


1.7 


7.1 


13.9 


25.1 


52.2 


j. Items that require students to interpret tabular or 
graphical data 


9.2 


28.8 


35.6 


20.3 


5.8 


k. Items that require the application of familiar 
concepts or principles to different or unfamiliar 
situations 


4.1 


9.2 


37.1 


32.0 


17.7 


1. Items that require a critique or analysis of a 


28.2 


17.7 


33.7 


11.2 


8.8 



suggested solution to a problem 
m. Other (specify) 



f 



Indicate the extent to which this course~as it is taught to the single designated class ~is interdisciplinary 
or integrated with another course, if at all. By this we mean that certain topics are taught together to 
enhance the depth of both, such as combining math and science units that rely on statistical analysis or 
using mathematical skills to represent and understand historical or political trends. Circle the • 

appropriate response on each line. 





Never 


1 or 2 
periods 
per 

semesrer 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 

period per 
week 


I use examples or illustrations from other 
subject areas in my math teaching, e.g., in 
"story" or other application problems. 


4.1 


19.0 


27.9 


28.9 


19.7 


I blend the teaching of another subject 
area into my math teaching in this class. 


19.9 


28.0 


28.4 


14.9 


8.8 


I work with teachers from one or more 
subject areas to integrate other subjects 
into my math teaching. 


50.2 


29.2 


10.5 


6.1 


4.1 



d. A subject area I use more often than others to illustrate math applications is 



Science 


12 


Science 


180 


Business 


1 


English/Language Arts 


5 


Other 


2 


Business 


35 






Computers 


*•» 






Other 


31 



III. Instruction 



Teacher activity: Different instructional strategies involve different types of teacher activity during a 
lesson. For each t eacher activity listed below, first circle the response that indicates how often you do 
it with your designated class, and then circle the letter that indicates about how much total time per 
period you typically spend when you do it. (You may engage in more than one activity during a class 
session.) Do not answer the "time" item if your "frequency" response is "never.” 



Frequency of Use 



1 or 2 
periods 





Never 


per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per 
week 


a. Lecture, perhaps occasionally soliciting 
brief student input or using the board or 
overhead projector to highlight a key 
term or present an outline 


2.7 


3.1 


5.8 


11.6 


76.9 


b. Demonstrate a concept, using two- 
dimensional graphics such as drawings 
on the board, overhead projector, or 
computer 


0.7 


2.7 


8.2 


17.7 


70.7 


c. Demonstrate a concept, using three- 
dimensional tools such as manipulatives, 
models, or other objects 


7.5 


24.8 


30.6 


22.1 


15.0 


d. Lead students in recitation, drills, or 
quest ion-and-answer sessions 


13.0 


18.8 


17.7 


19.8 


30.7 


e. Observe or monitor student-led whole 
class discussions or demonstrations 
related to mathematics 


26.0 


28.8 


21.2 


13.7 


10.3 


f. Provide individual or small group 
tutoring as needed during individual 
seatwork or small group activities 
involving everyone in class 


2.7 


7.5 


11.5 


23.4 


54.9 


g. Provide remedial or enriching 

instruction to a pull-out group while the 
rest of the class works on assignments 


31.2 


26.0 


19.2 


10.6 


13.0 


h. Work on administrative tasks, such as 
recordkeeping, while students work on 
assignments 


42.5 


19.9 


15.4 


13.4 


8.9 


i. Administer a test or a quiz 


0.0 


2.0 


36.7 


49.7 


11.2 


j. Stimulate student discussions of 
approaches to solving problems, 
explanations of their mathematical 
thinking, or open-ended questions 


. 1.4 


8.3 


20.4 


27.0 


42.9 


k. Demonstrate uses of technology in 
mathematics 


7.8 


23.7 


26.8 


19.3 


22.0 


1. Set up and monitor or supervise 
cooperative learning activities 


12.6 


21.1 


29.9 


18.0 


18.4 
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Time Per Typical rise 



<. 10 
minutes 


< '/2 
period 


A '/2 
period 


17.3 


57.0 


25.0 


24.2 


53.3 


21.8 


47.6 


13.5 


39.0 


53.2 


32.4 


14.4 


35.2 


39.8 


24.5 


20.8 


47.9 


30.6 


41.2 


43.2 


15.6 


64.0 


31.1 


4.9 


2.8 


17.4 


78.0 


25.4 


57.3 


16.5 


36.9 


41.8 


21.3 


9.9 


40.1 


49.6 



14. Context factors: Listed below are two kinds of factors that may characterize a class and, where 

present, may require accommodation in instruction. The first set of factors are student characteristics 
that contribute to diversity. The second set are usually viewed as problematic. Both types vary quite a 
lot across schools, districts, and regions. For each factor, please indicate whether it is present to any 
degree. 







Present in Class? 




Student Diversity 


No 


Minimally 


Moderately 


Considerably 


a. Students have very different 

academic abilities in this subject. 


2.7 


23.4 


44.7 


29.2 


b. Students have different backgrounds, 
e.g., economic, language, 
culture. 


9.1 


40.9 


29.7 


20.3 


c. Some students have special needs, 
e.g., physical or emotional 
impairment. 


30.1 


49.3 


13.2 


7.4 


Problems 










d. Students are inadequately prepared 
for level of material. 


15.2 


45.3 


27.7 


11.5 


e. Students are disinterested. 


16.2 


46.6 


25.3 


11.5 


f. There are threats to students’ 
personal safety. 


80.4 


17.6 


1.4 


0.7 


g. There is a shortage of appropriate 
technology for student use. 


33.2 


27.1 


24.1 


15.6 


h. There is a shortage of other 

equipment for student use. 


37.1 


32.7 


21.1 


9.2 


i. Students have low morale. 


34.2 


43.7 


15.6 


6.4 


j. Parents are not interested in their 
child’s learning. 


36.5 


38.5 


19.6 


5.4 


k. Family problems preoccupy students. 


13.9 


54.7 


23.6 


7.8 


1. There is a shortage of appropriate 
technology for teacher use. 


30.2 


32.5 


24.7 


12.5 


m. There is a shortage of other 

equipment for teacher use. 


32.9 


33.9 


21.7 


11.5 




33.6 



29.5 



14.2 
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n. There is a shortage of preparation 
and planning time for teachers. 



22.4 



14. 



Context factors (cont.): 



Problems (corn.) 

o. There are threats to teachers' 

personal safety. 

p. Physical facilities are inadequate. 

q. Administrative support for instruction 

is inadequate. 



Present in Class? 



No 


Minimally 


Moderately 


Considerably 


82.4 


14.9 


2.7 


0.0 


49.8 


30.8 


13.6 


5.8 


55.6 


31.2 


9.5 


3.7 
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Student activities: Listed below are many kinds of student lesson activities, some with a long history 
of use in math classes and others associated with recent reform efforts. First, scan the whole list from 
a to x to see the array of choices, then circle the number next to each that indicates how often— if at all- 
-the way you teach the designated class during the regular class period requires that students engage in 
each activity. Do not answer the "time" item if your "frequency" response is "never." 



Frequency of Use 



Time Per Tvnical Use 



1 or 2 





Never 


periods 

per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per 
week 


<_ 10 
minutes 


< '/2 
period 


-> '/2 
period 


a. Listen to or observe 
teacher presentations 


3.0 


1.7 


4.1 


7.8 


83.4 


17.3 


56.0 


26.1 


b. Work individually on 
exercises, worksheets, or 
workbooks 


3.7 


5.1 


9.2 


14.6 


67.5 


23.8 


53.2 


22.7 


c. Use a textbook for 
activities other than 
homework, e.g., reading 
or reference 


19.5 


15.7 


21.5 


20.1 


22.9 


59.1 


28.0 


12.9 


d. Use supplementary printed 
materials 


4.4 


9.5 


23.6 


30.1 


31.8 


26.8 


53.9 


18.9 


e. Participate in whole-class 
discussions 


2.0 


9.2 


18.6 


19.0 


51.2 


36.6 


49.5 


13.9 


f. Explain how what they 
learn in class relates to the 
real world 


7.8 


19.0 


31.6 


25.9 


15.6 


68.0 


27.5 


4.5 


g. Engage in on-task 

discussion primarily with 
other students 


5.8 


9.5 


15.3 


24.1 


45.2 


39.3 


45.1 


15.3 


h. Correct or review 

previous day's homework 


2.0 


1.4 


4.4 


7.1 


85.1 


56.8 


36.5 


6.0 


i. Work on projects/ 
assignments that take a 
week or more to finish 


42.0 


31.1 


15.4 


9.6 


2.0 


29.7 


27.9 


42.4 


j. Give or listen to other 
students give oral reports 


58.6 


28.1 


6.8 


4.7 


1.7 


36.4 


29.5 


34.1 


k. Solve problems for which 
there are several 
appropriate answers or 
approaches 


3.7 


15.6 


28.5 


24.4 


27.5 


36.7 


48.6 


14.7 


1. Write a report or paper 


60.7 


30.2 


7.8 


1.4 


0.0 


33.0 


32.2 


34.8 
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Student activities (cont.):... Indicate how often-if at all-the lesson activities you use with the designated 
class during the regular class period require that students : 



» 



fr 



» 



i 






m. Work in small groups 

n. Do lab or field work 

o. Evaluate other students' work 

p. Practice or drill on 
computational skills 

q. Work on assignments that will 
be due as homework on the 
next day 

r. Respond orally to questions 
testing recall 

s. Respond orally to open-ended 
questions 

t. Take notes 

u. Explain to the whole class 
solutions developed 
individually or in small groups 

v. Participate in structured 
cooperative learning activities 

w. Take tests, quizzes, other 
assessments 

x. Wait for completion of non- 
academic tasks, e.g., 
attendance, homework 
collection, behavior 
management, etc. 



Frequency of Use 





I or 2 

periods 

per 


1 or 2 


One 


Never 


semester 
(or half 
year) 


periods 

per 

month 


period 

per 

week 


3.4 


12.9 


20.3 


24.1 


56.7 


26.6 


11.9 


2.0 


38.6 


23.9 


14.7 


10.2 


16.9 


12.8 


20.3 


20.7 


6.8 


7.1 


8.2 


13.9 



3.1 


5.1 


12.6 


25.5 


6.6 


8.6 


16.2 


24.5 


3.7 


3.1 


7.1 


13.6 


12.6 


15.6 


22.8 


26.2 


16.9 


20.3 


29.0 


16.6 


1.4 


30.2 


49.8 


18.0 


20.5 


18.4 


8.3 


10.1 





Time Per TvDical 1 ko 


> 1 
period 
per 
week 


<. 10 
minutes 


< 'h 

period 


_> V2 
period 


39.3 


9.6 


52.5 


37.1 


2.0 


16.4 


28.9 


52.3 


12.6 


59.6 


31.5 


8.4 


29.0 


49.6 


36.4 


13.1 


63.9 


45.2 


40.4 


13.6 


53.4 


65.5 


28.1 


6.0 


43.8 


67.7 


26.0 


5.6 


72.4 


39.1 


44.5 


15.7 


22.8 


53.4 


36.0 


9.9 


17.2 


13.6 


46.2 


39.8 


0.7 


1.4 


25.1 


71.5 


42.7 


93.9 


5.2 


0.4 



» 



O 





± 



E-13 



Non-academic time: How many minutes during each meeting of your designated class do you 
typically spend on activities not directly related to the academic content of this course? (Examples: 
listening to or making announcements, taking attendance, establishing rapport, or handling disruptions.) 



25% = 2 
50% = 5 
75% = 6 



Demonstrating math competencies: As a result of successful study, students are able to demonstrate 
proficiency in tasks of different levels of difficulty and complexity. In the designated class how often 
do students have formal or informal opportunities to demonstrate the following competencies? Circle 
one response on each line. 



a. Memorize basic number facts 

or formulae 

b. Understand a math concept 

c. Collect original data 

d. Organize, summarize, or display 

information 

e. Use estimation to determine the 

reasonableness of results 

f. Perform computations using basic 

number facts 

g. Use algebraic or geometric reasoning 

strategies to solve problems 

h. Analyze and interpret information 

i. Generalize from patterns or examples 

j. Generate original examples 



1 or 2 
periods 



Never 


per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> I 
period 
per week 


6.8 


9.9 


24.3 


27.7 


31.2 


0.0 


0.0 


2.03 


21.1 


76.9 


21.7 


38.6 


24.5 


12.1 


3.1 


7.2 


23.5 


27.6 


20.8 


20.8 


3.7 


12.6 


23.5 


34.7 


25.5 


2.4 


4.4 


7.1 


17.0 


69.0 


0.3 


2.4 


9.2 


18.0 


70.1 


1.0 


7.8 


18.7 


29.6 


42.9 


2.7 


4.8 


24.9 


30.0 


37.5 


10.6 


26.3 


33.4 


19.8 


9.9 
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Case Study Reponses (N= 41 ) 

Teaching and Learning Conditions of the School and Designated 
Mathematics Class 



When you answer the questions, think of a single class for which you had primary responsibility last 
semester. If you were responsible for multiple sections of one or more of the focus courses, select your 
first such class of the day (not homeroom), and think of this as the designated, class . 

What is the course title? 



What is die usual schedule and length (in minutes) of daily class meetings for the designated 
c lass ? If the weekly schedule is normally the same, just complete Week 1 . 



Week I Week 2 

Monday 


Examples 


Wk 1 Wk 2 ! Wk 1 

i 

90 i 45 


Tuesday 


j 

• 90 ! 45 


Wednesday 


1 

90 i 45 


Thursday 


t 

90 i 45 


Friday 


1 

90 i 45 



In what grade(s) are the students in your designated class enrolled? Enter the number of 
students at each grade level. 



Ungraded 


9th 


6th 


10 th 


7th 


11th 


8th 


12th 


Estimate the percentage of students in your designated class who function at #»arh l^v^i r»f 
academic ability in mathematics. Use the best information you have readily available 
(including your own professional judgment). Percentages should total to 100%. 


Much above the SCHOOL average 


19.5 % 


Somewhat above the SCHOOL average 


27.2 % 


At the SCHOOL average 


35.3 % 


Somewhat below the SCHOOL average 


12.3 % 


Much below the SCHOOL average 


6.5 % 


TOTAL 


100 % 
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In your opinion, what percentage of the students who enrolled in this class at the beginning are 
capable of learning the required material given available time and resources? 



percent 



6. At this school, how much actual influence do you think teachers have over the following? 





No 

influence 








Complete 

influence 


a. Setting policy on grouping students in 
classes by ability 


19.5 


17.1 


24.4 


31.7 


7.3 


b. Deciding which students take which 
courses 


4.9 


12.2 


39.0 


41.5 


2.4 


c. Determining discipline policy 


0.0 


0.0 


4.9 


63.4 


31.7 



within your classroom 



7. How much control do you have over each of the following areas of planning and teaching? On 
the left response set, indicate how much control you have through participation or 
representation in department, or school, or district decision groups; on the right, indicate the 
control you have in your own classroom. 



Through my participation in 
department, school, 
or district decisions 



In my own classroom 



No 

Control 




Complete 

Control 




No 

Control 






Complete 

Control 


12.5 


5.0 


32.5 


40.0 


10.0 


a. Selecting textbooks 


12.2 


14.6 


7.3 


41.5 


24.4 


7.5 


7.5 


17.5 


52.5 


15.0 


b. Selecting instructional materials 


0 


0 


0 


48.8 


51.2 


12.5 


7.5 


25.0 


37.5 


17.5 


c. Selecting content, topics and skills to 
be taught 


2.4 


2.4 


9.8 


46.3 


39.0 


12.5 


5.0 


12.5 


20.0 


50.0 


d. Selecting teaching techniques 


0 


0 


0 


12.2 


87.8 


12.5 


12.5 


22.5 


15.0 


37.5 


e. Determining the amount of homework 
to be assigned 


0 


0 


2.4 


14.6 


82.9 


17.5 


2.5 


25.0 


35.0 


20.0 


f. Setting standards for grading in your 
classes 


0 


0 


2.4 


34.1 


63.4 


12.5 


12.5 


22.5 


32.5 


20.0 


g. Disciplining students 


0 


0 


4.9 


56.1 


39.0 



O 
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II. Course Content and Emphasis 

8. Math content areas: Estimate the percentage of time (if any) that you have sper 
during this course teaching each content area listed below in the designated class 
should equal 100 percent. 

Number Use/Operations 

(Estimation, computational algorithms, fractions, decimals, ratio and 
proportion, percent, real numbers, number theory, order of operations, 
relationships between operations) 

Measurement 

(Use of measuring instruments, theory [arbitrary, standard, size], 
conversions, metric [SI] system, length, perimeter, area, volume, surface area 
angles, circles [e.g., pi, radius, diameter, area], Pythagorean theorem, mass 
[weight], time, temperature, speed) 

Consumer Applications 

(Simple interest, compound interest, rates [e.g., discount, commission] 
spread sheets) 

Data Analysis 

(Bar graphs, histograms, pictographs, line graphs, stem and leaf plots scatter 
plots, box plots, mean, median, mode, mean deviation, smoothing of graphs) 

Pre-Algebra 

(Integers, absolute value, exponents, scientific notation, use of variables, 
expressions, evaluation of formulas and expressions, one-step equations 
coordinate plane) 

Basic Algebra 

(Multi-step equations, inequalities, literal equations, lines [slope and 
intercept], operations on polynomials, factoring, square roots and radicals, 
operations on radicals, rational expressions) 

Advanced Algehra 

(Quadratic equations, systems of equations, systems of inequalities, 
compound inequalities, matrices [determinants], conic sections, rational, 
negative exponents, radicals, rules for exponents, complex numbers binomial 
theorem, factor [remainder theorems], field properties of real number system) 

Functions 

(Notation, relations, linear, quadratic, polynomial, rational, logarithmic, 
exponential, trigonometric/circular, inverse, composition) 

Basic (non-mea surementi Geometry 

(Basic terminology, relationships between lines, angles, planes, triangles, 
quadrilaterals, polygons, congruence, similarity, parallels, circles, constrictions) 

Advanced Geometry 

(Logic, reasoning, proof, symmetries, loci, spheres, cones, cylinders 
polyhedra, 3-dimensional relationships, transformational, coordinate, vectors 
analytic, non-Euclidean, topology) 
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or will spend 
The total 



% 



% 



% 



% 



% 



% 



% 



% 



% 



% 



[Topic list continues on next page ] 



Math content areas (cont.): 



Trigonometry 

(Basic ratios, radian measures, right triangle trigonometry, law of Sines, 
Cosines, identities, trigonometric equations, polar coordinates, periodicity, 
amplitude) 



Statistics 

(Variability/standard deviation, quartiles, percentiles, bivariate distributions, 
sampling, confluence intervals, correlation, lines of best fit, hypothesis testing. 
Chi-square, data transformation, central limit theorem) 

Probability 

(Sample spaces, compound probability, conditional probability, independent 
and dependent events, empirical, expected value, binomial distribution, 
normal curve) 

Finite Math/Snecial Topics 

(Sets, logic, mathematical induction, linear programming, networks, iteration 
and recursion, permutations/combinations, simulations, fractals) 



Analysis 

(Sequences and series, limits, continuity, rates of change, maxima, minima, 
differentiation, integration) 



TOTAL 



107 
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Dimensions of math: Math courses approach the study of skills and concepts in many ways, and 
material covered during a given period may include several emphases. Listed below are cross- 
cutting dimensions of mathematics that may be covered during any class meeting. First, please 
circle the response that indicates how frequently activities in the designated class emphasize each 
dimension. Second, circle the response that indicates about how much time you typically spend on 
that dimension when you do include it. (For example, if you usually open class meetings with a 
short, real-life application problem, you would circle 4 for "more than once a week" and a for "10 
minutes or less.”) Do not answer the "time" item if your "frequency" response is "never." If your 
school is not on semesters, answer for half a year. 



Frequency of Use Time Per Typical Ike 

1 or 2 
periods 







Never 


per 

semester 
(or half 
year) 


Lesson 


content emphasis: 






a. 


Computational techniques 


5.0 


5.0 


b. 


Specific math facts or 
definitions 


0 


2.5 


c. 


Broad math concepts 


0 


10.0 


d. 


Problem solving/inquiry 
skills 


0 


2.5 


e. 


Skill in communicating in 
speech or writing about 
mathematical ideas or 
applications 


15.0 


10.0 


f. 


Importance of 
mathematics in daily life 


0 


12.5 


g- 


Applications of 
mathematics in science 


2.6 


30.8 


h. 


Applications of 
mathematics in business 
and industry 


0 


30.8 



1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per 
week 


<. 10 
minutes 


< 'h 

period 


>. '/i 

period 


7.5 


7.5 


75.0 


32.5 


40.0 


27.5 


12.5 


17.5 


67.5 


42.5 


42.5 


15.0 


17.5 


30.0 


42.5 


20.5 


46.2 


33.3 


12.5 


27.5 


57.5 


10.3 


38.5 


51.3 


17.5 


25.0 


32.5 


60.0 


34.3 


5.7 


32.5 


35.0 


20.0 


65.9 


31.7 


2.4 


30.8 


20.5 


15.4 


53.8 


41.0 


5.1 


43.6 


25.6 


0 


55.0 


37.5 


7.5 
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Objectives: During the year, lessons in a course vary in the difficulty and complexity of their 
objectives for student learning, and in the strategies used to achieve them. Listed below are typical 
learning objectives. Circle the response that indicates for each objective the average frequency 
with which it occurs in the designated class . 



Student learning objective: 

a. Memorize (e.g., facts, definitions, or formulae) 

b. Understand (e.g., concepts, relationships, 
theorems) 

c. Collect data (e.g., observe, measure, count) 

d. Order, compare, estimate, approximate 

e. Perform mathematical operations, execute 
algorithms, classify 

f. Solve "story" problems with familiar 
structures, replicate proofs 

g. Recognize and solve story problems 
with unfamiliar or complex structures 

h. Interpret data (e.g., charts, graphs, tables) 

i. Build and revise theories, develop proofs 



Never 


1 or 2 
periods 
per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per 
week 


0 


26.8 


14.6 


31.7 


26.8 


0 


0 


2.4 


22.0 


75.6 


12.5 


29.3 


46.3 


12.2 


0 


0 


28.2 


38.5 


20.5 


12.8 


0 


0 


5.0 


20.0 


75.0 


0 


9.8 


22.0 


34.1 


34.1 


0 


25.0 


22.5 


37.5 


15.0 


2.5 


27.5 


35.0 


17.5 


17.5 


34.1 


17.1 


22.0 


19.5 


7.3 



11 . 



Assessment content: On the tests, quizzes, projects, portfolio products, and other formal 
assessment strategies you use in this class, how often do the following kinds of items appear? 



ft 



ft 



ft 



ft 



ft 



ft 



Only during 





Never 


units 

focusing on 
that skill 


Occasionally 


Average of 
once per 
assessment 


Twice or 
more per 
assessment 


a. Items that require students to compute answers 


0 


2.4 


0 


2.4 


95.1 


b. Items that require students to recognize or recall 
definitions or concepts 


2.5 


2.5 


17.5 


25.0 


52.5 


c. Items that allow more than one solution or 
approach 


0 


2.4 


34.1 


46.3 


17.1 


d. Items that require more than one step to reach a 
solution 


0 


0 


0 


4.9 


95.1 


e. Items that require students to explain concepts in 
words or pictures 


2.4 


17.1 


34.1 


31.7 


14.6 


f. Items that require students to observe or measure 


20.0 


17.5 


45.0 


12.5 


5.0 


g. Items that require students to compare an 
estimated answer with a calculated or computed 
answer 


14.6 


31.7 


41.5 


7.3 


4.9 


h. Items that require the use of algorithms to solve 
problems 


4.9 


4.9 


17.1 


43.9 


29.3 


i. Items that require students to describe how they 
solved problems by showing all their work, 
writing explanations, or drawing pictures 


0 


0 


12.2 


31.7 


56.1 


j . Items that require students to interpret tabular or ' 
graphical data 


14.6 


17.1 


39.0 


22.0 


7.3 


k. Items that require the application of familiar 
concepts or principles to different or unfamiliar 
situations 


0 


4.9 


43.9 


36.6 


14.6 


1. Items that require a critique or analysis of a 


29.3 


22.0 


26.8 


19.5 


2.4 



suggested solution to a problem 
m. Other (specify) 



ft 
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Indicate the extent to which this course— as it is taught to the single designated class — is interdisciplinary 
or integrated with another course, if at all. By this we mean that certain topics are taught together to 
enhance the depth of both, such as combining math and science units that rely on statistical analysis or 
using mathematical skills to represent and understand historical or political trends. Circle the 
appropriate response on each line. 





Never 


I or 2 
periods 
per 

semester 


I or 2 
periods 
per 
mouth 


One 

period 

per 

week 


> 1 

period per 
week 


a. I use examples or illustrations from other 
subject areas in my math teaching, e.g., in 
"story" or other application problems. 


2.5 


22.5 


35.0 


22.5 


17.5 


b. I blend the teaching of another subject 
area into my math teaching in this class. 


15.0 


42.5 


25.0 


10.0 


7.5 


c. I work with teachers from one or more 


39.0 


41.5 


7.3 


4.9 


7.3 



subject areas to integrate other subjects 
into my math teaching. 

d. A subject area I use more often than others to illustrate math applications is 



Science 5 



Science 23 

English/Language Arts 1 

Business 6 

Other 6 



III. Instruction 



Teacher activity: Different instructional strategies involve different types of teacher activity during a 
lesson. For each teacher activity listed below, first circle the response that indicates how often you do 
it with your designated class, and then circle the letter that indicates about how much total time per 
period you typically spend when you do it. (You may engage in more than one activity during a class 
session.) Do not answer the "time" item if your "frequency” response is "never." 



Frequency of Use 



Time Per Typical Use 





Never 


1 or 2 

periods 

per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per 
week 


<. 10 
minutes 


< */2 

period 


>L */2 

period 


a. Lecture, perhaps occasionally soliciting 
brief student input or using the board or 
overhead projector to highlight a key 
term or present an outline 


0 


2.4 


0 


4.9 


92.7 


10.0 


57.5 


32.5 


b. Demonstrate a concept, using two- 
dimensional graphics such as drawings 
on the board, overhead projector, or 
computer 


5.0 


0 


10.0 


17.5 


67.5 


20.5 


64.1 


15.4 


c. Demonstrate a concept, using three- 
dimensional tools such as manipulaiives, 
models, or other objects 


15.4 


33.3 


25.6 


20.5 


5.1 


43.8 


46.9 


9.4 


d. Lead students in recitation, drills, or 
question-and-answer sessions 


12.8 


15.4 


20.5 


20.5 


30.8 


62.9 


28.6 


8.6 


e. Observe or monitor student-led whole 
class discussions or demonstrations 
related to mathematics 


29.3 


24.4 


29.3 


2.4 


14.6 


14.8 


55.6 


29.6 


f. Provide individual or small group 
tutoring as needed during individual 
seatwork or small group activities 
involving everyone in class 


0 


4.9 


19.5 


7.3 


68.3 


19.5 


41.5 


39.0 


g. Provide remedial or enriching 

instruction to a pull-out group while the 
rest of the class works on assignments 


38.5 


25.6 


15.4 


12.8 


7.7 


37.5 


50.0 


12.5 


h. Work on administrative tasks, such as 
recordkeeping, while students work on 
assignments 


20.5 


35.9 


15.4 


10.3 


17.9 


67.7 


25.8 


6.5 


i. Administer a test or a quiz 


2.4 


4.9 


48.8 


36.6 


7.3 


2.5 


7.5 


90.0 


j. Stimulate student discussions of 
approaches to solving problems, 
explanations of their mathematical 
thinking, or open-ended questions 


4.9 


9.8 


19.5 


34.1 


31.7 


25.6 


46.2 


28.2 


k. Demonstrate uses of technology in 
mathematics 


12.2 


24.4 


19.5 


17.1 


26.8 


33.3 


33.3 


33.3 


1. Set up and monitor or supervise 
cooperative learning activities 


24.4 


22.0 


29.3 


12.2 


12.2 


12.9 


29.0 


58.1 
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Context factors: Listed below are two kinds of factors that may characterize a class and, where 
present, may require accommodation in instruction. The first set of factors are student characteristics 
that contribute to diversity. The second set are usually viewed as problematic. Both types vary quite a 
lot across schools, districts, and regions. For each factor, please indicate whether it is present to any 
degree. 



Present in Class*? 



Student Diversity 


No 


Minimally 


Moderately 


Considerably 


a. Students have very different 

academic abilities in this subject. 


0 


24.4 


43.9 


31.7 


b. Students have different backgrounds, 
e.g., economic, language, 
culture. 


12.5 


30.0 


40.0 


17.5 


c. Some students have special needs, 
e.g., physical or emotional 
impairment. 


17.1 


61.0 


17.1 


4.9 


Problems 










d. Students are inadequately prepared 
for level of material. 


19.5 


39.0 


26.8 


14.6 


e. Students are disinterested. 


12.2 


56.1 


22.0 


9.8 


f. There are threats to students’ 
personal safety. 


87.5 


12.5 


0 


0 


g. There is a shortage of appropriate 
technology for student use. 


25.0 


35.0 


30.0 


10.0 


h. There is a shortage of other 

equipment for student use. 


32.5 


32.5 


25.0 


10.0 


i. Students have low morale. 


47.5 


30.0 


20.0 


2.5 


j. Parents are not interested in their 
child’s learning. 


42.5 


27.5 


27.5 


2.5 


k. Family problems preoccupy students. 


10.0 


60.0 


22.5 


7.5 


1. There is a shortage of appropriate 
technology for teacher use. 


30.8 


30.8 


33.3 


5.1 


m. There is a shortage of other 

equipment for teacher use. 


27.5 


35.0 


. 32.5 


5.0 


n. There is a shortage of preparation 

and planning time for teachers. 


26.8 


29.3 


29.3 


14.6 
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Context factors (cont.): 



Present in Class? 





No 


Minimally 


Moderately 


Considerably 


Problems Icont.) 










o. There are threats to teachers’ 
personal safety. 


77.5 


22.5 


0 


0 


p. Physical facilities are inadequate. 


35.9 


41.0 


17.9 


5.1 


q. Administrative support for instruction 
is inadequate. 


50.0 


30.0 


17.5 


2.5 



O 
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Student activities: Listed below are many kinds of student lesson activities, some with a long history 
of use in math classes and others associated with recent reform efforts. First, scan the whole list from 
a to x to see the array of choices, then circle the number next to each that indicates how often— if at all- 
-the way you teach the designated class during the regular class period requires that students engage in 
each activity. Do not answer the "time" item if your "frequency" response is "never." 



Frequency of Use 



Time Per Typical Ike 





Never 


1 or 2 

periods 

per 

semester 
(or half 
year) 


a. Listen to or observe 
teacher presentations 


0 


0 


b. Work individually on 
exercises, worksheets, or 
workbooks 


2.6 


0 


c. Use a textbook for 
activities other than 
homework, e.g., reading 
or reference 


30.8 


15.4 


d. Use supplementary printed 
materials 


2.5 


15.0 


e. Participate in whole-class 
discussions 


10.0 


17.5 


f. Explain how what they 
learn in class relates to the 
real world 


15.0 


22.5 


g. Engage in on-task 

discussion primarily with 
other students 


12.5 


15.0 


h. Correct or review 

previous day's homework 


2.5 


2.5 


i. Work on 

projects/assignments that 
take a week or more to 
finish 


41.0 


30.8 


j. Give or listen to other 
students give oral reports 


51.3 


38.5 


k. Solve problems for which 
there are several 
appropriate answers or 
approaches 


7.5 


17.5 


1. Write a report or paper 


56.4 


41.0 



1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per 
week 


<_ 10 
minutes 


< */2 
period 


>_ '/z 

period 


0 


0 


100.0 


7.5 


60.0 


32.5 


10.3 


17.9 


69.2 


21.1 


42.1 


36.8 


7.7 


17.9 


28.2 


32.1 


46.4 


17.9 


27.5 


25.0 


30.0 


15.4 


48.7 


35.9 


12.5 


12.5 


47.5 


33.3 


52.8 


13.9 


22.5 


30.0 


10.0 


57.6 


36.4 


6.1 


12.5 


17.5 


42.5 


40.0 


48.6 


11.4 


5.0 


12.5 


77.5 


53.8 


43.6 


2.6 


20.5 


7.7 


0 


31.8 


18.2 


50.0 


10.3 


0 


0 


16.7 


22.2 


61.1 


40.0 


17.5 


17.5 


27.8 


52.8 


19.4 


2.6 


0 


0 


23.5 


29.4 


47.1 
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Student activities (cont.):... Indicate how often— if at all— the lesson activities you use with the designated 
class during the regular class period require that students : ~ ~ 



ft 



ft 



ft 



ft 



ft 



» 



ft 



ft 



Frequency of Use 



Time Per Typical TIsp 



1 or 2 
periods 





Never 


per 

semester 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per 
week 


<_ 10 
minutes 


< V4 

period 


_> 'A 
period 


m. Work in small groups 


12.5 


15.0 


12.5 


15.0 


45.0 


11.1 


41.7 


47.2 


n. Do lab or field work 


55.3 


18.4 


26.3 


0 


0 


11.8 


52.9 


35.3 


o. Evaluate other students' work 


33.3 


25.6 


15.4 


20.5 


5.1 


53.6 


39.3 


7.1 


p. Practice or drill on 
computational skills 


20.5 


17.9 


25.6 


15.4 


20.5 


60.6 


30.3 


9.1 


q. Work on assignments that will 
be due as homework on the 
next day 


0 


5.0 


12.5 


22.5 


60.0 


46.3 


31.7 


22.0 


r. Respond orally to questions 
testing recall 


2.5 


5.0 


2-5 


35.0 


55.0 


55.0 


37.5 


7.5 


s. Respond orally to open-ended 
questions 


5.0 


2.5 


20.0 


32.5 


40.0 


59.0 


35.9 


5.1 


t. Take notes 


5.0 


2.5 


5.0 


7.5 


80.0 


25.6 


48.7 


25.6 


u. Explain to the whole class 
solutions developed 
individually or in small groups 


15.0 


17.5 


30.0 


25.0 


12.5 


44.1 

% 


44.1 


11.8 


v. Participate in structured 
cooperative learning activities 


27.5 


20.0 


27.5 


7.5 


17.5 


6.7 


43.3 


50.0 


w. Take tests, quizzes, other 
assessments 


0 


0 


46.3 


31.7 


22.0 


2.4 


4.9 


92.7 


x. Wait for completion of non- 


12.8 


10.3 


0 


15.4 


61.5 


94.3 


2.9 


’■ 2.9 



academic tasks, e.g., 
attendance, homework 
collection, behavior 
management, etc. 
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Non-academic time: How many minutes during each meeting of your designated class do you 
typically spend on activities not directly related to the academic content of this course? (Examples: 
listening to or making announcements, taking attendance, establishing rapport, or handling disruptions.) 



25% = 3 
50% = 5 
75% = 8 



17. Demonstrating math competencies: As a result of successful study, students are able to demonstrate 
proficiency in tasks of different levels of difficulty and complexity. In the designated class , how often 
do students have formal or informal opportunities to demonstrate the following competencies? Circle 
one response on each line. 





Never 


1 or 2 
periods 
per 

scmcsicr 
(or half 
year) 


1 or 2 
periods 
per 
monih 


One 

period 

per 

week 


> 1 
period 
per week 


a. Memorize basic number facts 
or formulae 


9.8 


4.9 


22.0 


31.7 


31.7 


b. Understand a math concept 


0 


0 


2.4 


7.3 


90.2 


c. Collect original data 


37.5 


30.0 


25.0 


5.0 


2.5 


d. Organize, summarize, or display 
information 


17.9 


33.3 


20.5 


15.4 


12.8 


e. Use estimation to determine the 
reasonableness of results 


5.0 


25.0 


20.0 


32.5 


17.5 


f. Perform computations using basic 
number facts 


0 


7.3 


7.3 


9.8 


75.6 


g. Use algebraic or geometric reasoning 
strategies to solve problems 


0 


0 


4.9 


17.1 


78.0 


h. Analyze and interpret information 


7.3 


4.9 


19.5 


34.1 


34.1 


i. Generalize from patterns or examples 


2.4 


7.3 


22.0 


41.5 


26.8 


j. Generate original examples 


22.0 


41.5 


24.4 


12.2 


0 
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IV. Materials 

18. Teacher materials: Indicate the extent to which the materials and equipment listed below are 
available, and the frequency with which you use them to teach your designated class . Indicate 

frequency of use only if the item was available. (Note, a separate item will ask how often students use 
them.) 

0 = The item was not available 

1 = Item was available in limited supply or with limited functions so that use was less 

effective than it could have been. 

2 = Item was available in adequate supply and/or with appropriate functions for 

most effective use. 



Availability Frequency of Usp 

I or 2 





Not 

available 


Limited 

availability 


Full 

availability 


Never 


period 5 
per 

semester 
(or half 
year) 


1 or 2^ 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per week 


a. Board 


0 


2.4 


97.6 


2.6 


0 


5.1 


2.6 


89.7 


b. Graph paper 


0 


22.0 


78.0 


0 


17.5 


35.0 


30.0 


17.5 


c. Protractors, rulers, or compasses 


10.0 


10.0 


80.0 


8.3 


25.0 


36.1 


8.3 


22.2 


d. Appropriate calculator 


0 


12.2 


87.8 


2.5 


5.0 


7.5 


17.5 


67.5 


e. Overhead projector 


4.9 


14.6 


80.5 


10.3 


15.4 


15.4 


20.5 


38.5 


f. Film or videotape 


25.6 


46.2 


28.2 


30.0 


50.0 


20.0 


0 


0 


g. Computer or computer programs 


22.5 


65.0 


12.5 


30.3 


45.5 


18.2 


3.0 


3.0 


h. Manipulates, models, or other 
objects 

i. Other (specify) 


9.8 


53.7 


36.6 


10.5 


31.6 


39.5 


13.2 


5.3 
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19. Student materials: Indicate the extent to which materials and equipment listed below are 
available, and the frequency with which students in your designated class use them during a 
regular class period. Indicate frequency of use only if the item was available. 

0 = The item was not available 

1 = Item was available in limited supply or with limited functions so that use was less 

effective than it could have been. 

2 = Item was available in adequate supply and/or with appropriate functions for 

most effective use. 



Availability 



Frequency of Use 





Not 


Limited 




Available 


Availability 


a. Board 


2.5 


7.5 


b. Graph paper 


2.6 


23.1 


c. Protractors, rulers, or 


7.9 


18.4 


compasses 






d. Appropriate calculator 


0 


25.0 


e. Overhead projector 


10.3 


20.5 


f. Film or videotape 


40.5 


40.5 


g. Computer or computer 


30.8 


64.1 


programs 






h. Manipulatives, models, or 


23.1 


53.8 



other objects 



i. Other (specify) 



full 

Availability 


Never 


1 or 2 
periods 
per 

scnKStcr 
(or half 
year) 


1 or 2 
periods 
per 
month 


One 

period 

per 

week 


> 1 
period 
per 
week 


90.0 


5.3 


13.2 


34.2 


13.2 


34.2 


74.4 


0 


10.5 


36.8 


26.3 


26.3 


73.7 


5.6 


25.0 


30.6 


16.7 


22.2 


75.0 


0 


2.5 


0 


25.0 


72.5 


69.2 


37.8 


16.2 


21.6 


10.8 


13.5 


18.9 


62.1 


27.6 


10.3 


0 


0 


5.1 


35.5 


38.7 


19.4 


3.2 


3.2 


23.1 


9.1 


42.4 


33.3 


12.1 


3.0 
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The following hems concern Che questionnaire itself. Information you provide will help us improve the 
questionnaire for future use. F 



20. Were any of the questions confusing or unclear? 

No 

Yes If yes, please list the question number and describe the source of confusion. 

Number Source of confusion 



21. Use the space below to describe any other problems or make any recommendations about the 
questionnaire. 



Please remember to note the time you finish and record the total time you spent on this survey on the 
front page. Thank you again for your help. 
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Appendix G 
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Case Study Teacher and Observer Log Use 



ft 



ft 



ft 



ft 
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Frequency of Log Use 



ft 



ft 



ft 






ft 



ft 



ft 



ft 





Percent 

Total 

Observation 
Log use 


Percent 
Observer 
Observation 
Log use 
(n=41) 


Percent 
Teacher 
Observation 
Log use 
(n=41) 


Percent 
Case Study 
Teacher 
Log Use 
(n=700) 




(a) 


(b) 


(C) 


(d) 


a 


51.2 


39.0 


63.4 


40.3 


b 


92.7 


85.4 


100.0 


83.6 


c 


6.1 


2.4 


9.8 


6.6 


d 


7.3 


7.3 


7.3 


19.6 


e 


85.4 


85.4 


85.4 


79.3 


f 


75.6 


75.6 


75.6 


68.1 


g 


51.2 


41.5 


61.0 


51.1 


h 


15.9 


7.3 


24.4 


19.3 


i 


15.9 


7.3 


24.4 


10.1 


a 


98.8 


97.6 


100.0 


80.9 


b 


51.2 


46.3 


56.1 


34.1 


c 


9.8 


9.8 


9.8 


4.7 


d 


76.8 


70.7 


82.9 


50.4 


e 


15.9 


7.3 


24.4 


18.1 


f 


73.2 


70.7 


75.6 


57.4 


g 


2.4 


0.0 


4.9 


8.9 


h 


64.6 


61.0 


68.3 


52.9 


i 


8.5 


4.9 


12.2 


21.4 


j 


48.8 


29.3 


68.3 


33.7 


k 


11.0 


2.4 


19.5 


14.9 


1 


15.9 


9.8 


22.0 


13.4 


a 


100.0 


100.0 


100.0 


90.6 


b 


81.7 


80.5 


82.9 


59.9 


c 


35.4 


34.1 


36.6 


27.7 


d 


37.8 


34.1 


41.5 


29.3 


e 


53.7 


43.9 


63.4 


44.3 


f 


14.6 


7.3 


22.0 


18.6 


g 


46.3 


43.9 


48.8 


31.0 


h 


54.9 


48.8 


61.0 


51.3 


i 


3.7 


2.4 


4.9 


5.1 


j 


7.3 


7.3 


7.3 


3.6 


k 


41.5 


34.1 


48.8 


30.7 


1 


1.2 


2.4 


0.0 


1.4 


m 


40.2 


36.6 


43.9 


36.6 


n 


0.0 


0.0 


0.0 


2.0 


0 


28.0 


24.4 


31.7 


25.6 


P 


32.9 


17.1 


48.8 


30.9 


q 


53.7 


51.2 


56.1 


42.4 
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Frequency of Log Use 





Percent 

Total 

Observation 
Log use 


Percent 
Observer 
Observation 
Log use 
(n=41) 


Percent 
Teacher 
Observation 
Log use 
(n=41) 


Percent 
Case Study 
Teacher 
Log Use 
(n=700) 




(a) 


(b) 


(C) 


(d) 


r 


87.8 


85.4 


90.2 


40.4 


s 


59.8 


63.4 


56.1 


31.9 


t 


61.0 


53.7 


68.3 


47.9 


u 


36.6 


39.0 


34.1 


17.4 


V 


12.2 


4.9 


19.5 


11.7 


w 


8.5 


7.3 


9.8 


22.6 


X 


56.1 


48.8 


63.4 


9.4 


a 


78.0 


78.0 


78.0 


72.1 


b 


6.1 


2.4 


9.8 


15.3 


c 


6.1 


4.9 


7.3 


10.6 


d 


26.8 


17.1 


36.6 


36.4 


e 


37.8 


36.6 


39.0 


33.4 


f 


0.0 


0.0 


0.0 


1.4 


g 


0.0 


0.0 


0.0 


0.9 


h 


14.6 


12.2 


17.1 


8.6 


a 


12.2 


12.2 


12.2 


24.3 


b 


15.9 


19.5 


12.2 


22.1 


c 


13.4 


9.8 


17.1 


14.3 


d 


70.7 


70.7 


70.7 


70.6 


e 


1.2 


0.0 


2.4 


5.3 


f 


0.0 


0.0 


0.0 


1.6 


g 


0.0 


0.0 


0.0 


2.0 


h 


9.8 


4.9 


14.6 


7.7 
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Appendix H 

Comparing Case Study Teacher Log Entries with Observer Entries 



Agreement on occurrence Agreement on length of time 





Percent 

Agreement 


Where 

nonagreement, 
% of teachers 
reporting use 
(0 says no) 


Direct 

Agreement 


Within one 
response 
category 


Where 

nonagreement, 
% of teachers 
reporting 
more time 


item 


(a) 


(b) 


(C) 


(d) 


(e) 


10 a 


65.9 


85.7 


64.3 


85.7 


60.0 


b 


85.4 


100.0 


57.1 


94.3 


40.0 


c - 


92.7 


100.0 


0.0 


0.0 


100.0 


d 


75.6 


100.0 


100.0 


100.0 


NA 


e 


90.2 


100.0 


48.6 


85.7 


22.2 


f 


78.0 


66.7 


50.0 


75.0 


14.3 


g 


70.7 


83.3 


60.0 


93.3 


50.0 


h 


78.0 


88.9 


50.0 


100.0 


0.0 


i 


73.2 


81.8 


100.0 


100.0 


NA 


13 a 


97.6 


100.0 


77.5 


100.0 


44.4 


b 


75.6 


70.0 


68.4 


73.7 


100.0 


c 


100.0 


NA 


25.0 


100.0 


66.7 


d 


75.0 


70.0 


61.5 


92.3 


20.0 


e 


77.5 


88.9 


50.0 


100.0 


0.0 


f 


95.0 


100.0 


72.4 


96.6 


62.5 


g 


95.0 


100.0 


0.0 


0.0 


NA 


h 


87.5 


80.0 


92.0 


100.0 


100.0 


i 


92.5 


100.0 


0.0 


100.0 


50.0 


j 


55.0 


94.4 


54.5 


100.0 


80.0 


k 


82.5 


100.0 


100.0 


100.0 


0.0 


1 


82.5 


85.7 


66.7 


100.0 


100.0 


15 a 


100.0 


NA 


78.0 


100.0 


22.2 


b 


92.7 


66.7 


53.1 


100.0 


40.0 


c 


82.9 


57.1 


72.7 


90.9 


33.3 


d 


82.9 


71.4 


58.3 


91.7 


60.0 


e 


56.1 


72.2 


69.2 


92.3 


50.0 


f 


80.5 


87.5 


100.0 


100.0 


NA 


g 


75.6 


60.0 


50.0 


85.7 


50.0 


h 


73.2 


72.7 


76.5 


100.0 


0.0 


i 


92.7 


66.7 


0.0 


0.0 


0.0 


j 


95.1 


50.0 


50.0 


100.0 


100.0 


k 


70.7 


75.0 


54.5 


100.0 


60.0 


1 


97.6 


0.0 


0.0 


0.0 


0.0 


m 


92.7 


100.0 


53.3 


93.3 


57.1 


n 


100.0 


NA 


0.0 


0.0 


0.0 


0 


78.0 


66.7 


57.1 


85.7 


0.0 


P 


63.4 


93.3 


66.7 


100.0 


0.0 
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Comparing Case Study Teacher Log Entries with Observer Entries 



Agreement on occurrence 



Agreement on length of time 





Percent 

Agreement 


Where 

nonagreement, 
% of teachers 
reporting use 
(0 says no) 


Direct 

Agreement 


Within one 
response 
category 


Where 

nonagreement, 
% of teachers 
reporting 
more time 




(a) 


(b) 


(C) 


(d) 


(e) 


q 


85.4 


66.7 


68.4 


100.0 


50.0 


r 


80.5 


62.5 


71.0 


93.5 


33.3 


s 


82.9 


28.6 


71.4 


95.2 


0.0 


t 


80.5 


87.5 


42.9 


100.0 


50.0 


u 


80.5 


37.5 


72.7 


90.9 


33.3 


V 


65.9 


92.9 


0.0 


0.0 


0.0 


w 


97.6 


100.0 


100.0 


100.0 


NA 


X 


70.7 


75.0 


94.1 


100.0 


100.0 


a 


100.0 


NA 


68.8 


96.9 


0.0 


b 


92.7 


100.0 


100.0 


100.0 


50.0 


c 


97.6 


100.0 


50.0 


100.0 


0.0 


d 


80.5 


100.0 


71.4 


71.4 


0.0 


e 


97.6 


100.0 


60.0 


93.3 


66.7 


f 


97.6 


NA 


NA 


NA 


0.0 


g 


100.0 


NA 


NA 


NA 


100.0 


h 


95.1 


100.0 


60.0 


100.0 


100.0 


a 


94.9 


50.0 


100.0 


100.0 


0.0 


b 


92.3 


100.0 


20.0 


80.0 


50.0 


c 


92.3 


100.0 


75.0 


75.0 


0.0 


d 


89.7 


50.0 


63.0 


88.9 


50.0 


e 


97.4 


100.0 


0.0 


0.0 


0.0 


f 


100.0 


NA 


NA 


NA 


NA 


g 


100.0 


NA 


NA 


NA 


NA 


h 


89.7 


100.0 


100.0 


100.0 


0.0 
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Comparing Case Study Teacher Questionnaire Responses with Log Entries 



Comparing Teacher Responses to Comparing Teacher Responses to 

Questionnaire #1 with Log Entries Questionnaire #2 with Log Entries 



Questionnaire responses Questionnaire responses 

validated by log frequencies validated by log frequencies 





Direct 

Agreement 


Within one 
response 
category 


Where 

nonagreement, 
% of teachers 
underreporting 
on survey 


Direct 

Agreement 


Within one 
response 
category 


Where 

nonagreement, 
% of teachers 
underreporting 
on survey 




(a) 


(b) 


(C) 


(d) 


(e) 


(f> 


a 


41.5 


41.5 


66.7 


35.0 


35.0 


69.2 


b 


75.6 


75.6 


100.0 


75.0 


75.0 


100.0 


c 


51.2 


56.1 


35.0 


57.9 


89.5 


12.5 


d 


35.9 


46.2 


61.9 


63.2 


63.2 


100.0 


e 


67.5 


75.0 


80.0 


45.0 


50.0 


81.8 


f 


34.1 


41.5 


91.7 


25.0 


30.0 


86.7 


g 


22.5 


32.5 


85.2 


31.6 


42.1 


76.9 


h 


37.5 


65.0 


63.6 


45.0 


65.0 


60.0 


i 


61.0 


75.6 


55.6 


52.6 


63.2 


33.3 


al 


78.0 


80.5 


50.0 


89.5 


89.5 


50.0 


bl 


50.0 


57.5 


15.0 


40.0 


55.0 


33.3 


cl 


56.4 


71.8 


17.6 


47.4 


84.2 


10.0 


dl 


35.9 


48.7 


92.0 


35.0 


45.0 


76.9 


el 


46.3 


68.3 


54.5 


57.9 


68.4 


100.0 


fl 


65.9 


70.7 


71.4 


55.0 


70.0 


88.9 


gl 


53.8 


74.4 


55.6 


68.4 


94.7 


50.0 


hi 


23.1 


28.2 


90.0 


36.8 


42.1 


91.7 


il 


41.5 


58.5 


62.5 


45.0 


60.0 


90.9 


jl 


31.7 


61.0 


60.7 


25.0 


40.0 


73.3 


kl 


41.5 


56.1 


37.5 


42.1 


73.7 


30.0 


11 


48.8 


65.9 


42.9 


60.0 


75.0 


37.5 


a2 


58.5 


95.1 


37.5 








b2 


41.5 


82.9 


36.4 








c2 


17.1 


41.5 


8.0 








d2 


51.2 


82.9 


57.1 








e2 


22.0 


48.8 


11.1 








f2 


46.3 


92.7 


36.4 








g2 


12.2 


41.5 


16.7 








h2 


41.5 


56.1 


42.9 








i2 


63.4 


87.8 


28.6 








j2 


34.1 


87.8 


72.0 








k2 


22.0 


65.9 


22.2 








12 


26.8 


51.2 


20.0 








al 


100.0 


100.0 


0.0 


95.0 


95.0 


100.0 


bl 


74.4 


79.5 


80.0 


63.2 


78.9 


42.9 



ERiC 
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Comparing Case Study Teacher Questionnaire Responses with Log Entries 



Comparing Teacher Responses to Comparing Teacher Responses to 

Questionnaire #1 with Log Entries Questionnaire #2 with Log Entries 




item 



15 



Questionnaire responses Questionnaire responses 

validated by log frequencies validated by log frequencies 





Direct 

Agreement 


Within one 
response 
category 


Where 

nonagreement, 
% of teachers 
underreporting 
on survey 


Direct 

Agreement 


Within one 
response 
category 


Where 

nonagreement, 
% of teachers 
underreporting 
on survey 




(a) 


(b) 


(c) 


(d) 


(e) 


(o 


cl 


41.0 


53.8 


65.2 


60.0 


75.0 


62.5 


dl 


40.0 


47.5 


58.3 


45.0 


65.0 


63.6 


el 


47.5 


50.0 


71.4 


42.1 


47.4 


72.7 


fl 


37.5 


60.0 


68.0 


36.8 


57.9 


75.0 


gi 


40.0 


50.0 


62.5 


10.5 


21.1 


70.6 


hi 


72.5 


82.5 


45.5 


78.9 


78.9 


50.0 


il 


64.1 


76.9 


42.9 


73.7 


84.2 


60.0 


jl 


84.6 


94.9 


50.0 


89.5 


94.7 


50.0 


kl 


37.5 


55.0 


68.0 


47.4 


63.2 


60.0 


11 


87.2 


97.4 


80.0 


89.5 


100.0 


100.0 


ml 


62.5 


75.0 


65.2 


45.0 


60.0 


63.6 


nl 


76.3 


97.4 


58.3 


89.5 


94.7 


0.0 


ol 


33.3 


53.8 


71.4 


38.9 


50.0 


90.9 


pl 


28.2 


46.2 


68.0 


36.8 


42.1 


50.0 


qi 


57.5 


75.0 


62.5 


47.4 


57.9 


60.0 


rl 


42.5 


57.5 


45.5 


60.0 


65.0 


75.0 


Si 


40.0 


60.0 


42.9 


45.0 


55.0 


81.8 


tl 


70.0 


82.5 


50.0 


65.0 


75.0 


42.9 


ul 


47.5 


• 57.5 


68.0 


61.1 


88.9 


85.7 


vl 


57.5 


87.5 


80.0 


65.0 


90.0 


28.6 


wl 


41.5 


56.1 


66.7 


35.0 


55.0 


61.5 


xl 


38.5 


53.8 


33.3 


71.4 


71.4 


50.0 


a2 


43.9 


95.1 


40.9 








b2 


46.3 


90.2 


20.0 








c2 


12.2 


48.8 


34.8 








d2 


34.1 


92.7 


80.0 








e2 


31.7 


78.0 


43.5 








f2 


■ 26.8 


58.5 


27.3 








g2 


26.8 


58.5 


54.2 








h2 


61.0 


92.7 


28.6 








i2 


4.9 


22.0 


15.0 








j2 


24.4 


36.6 


87.5 








k2 


34.1 


73.2 


22.7 








12 


0.0 


9.8 


5.9 








m2 


29.3 


65.9 


33.3 








n2 


9.8 


17.1 


0.0 








o2 


34.1 


56.1 


21.4 
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Comparing Case Study Teacher Questionnaire Responses with Log Entries 



Comparing Teacher Responses to Comparing Teacher Responses to 

Questionnaire #1 with Log Entries Questionnaire #2 with Log Entries 



Questionnaire responses 
validated by log frequencies 



Questionnaire responses 
validated by log frequencies 



item 



18 



19 





Direct 

Agreement 


Within one 
response 
category 


Where 

nonagreement, 
% of teachers 
underreporting 
on survey 


Direct 

Agreement 


Within one 
response 
category 


Where 

nonagreement, 
% of teachers 
underreporting 
on survey 




(a) 


(b) 


(c) 


(d) 


(e) 


(o 


P2 


46.3 


68.3 


35.7 








q2 


36.6 


95.1 


50.0 








r2 


43.9 


97.6 


26.1 








s2 


43.9 


82.9 


28.6 








t2 


46.3 


95.1 


40.0 








u2 


31.7 


68.3 


28.6 








v2 


19.5 


41.5 


9.1 








w2 


78.0 


95.1 


22.2 








x2 


68.3 


80.5 


14.3 








al 


84.6 


87.2 


33.3 


85.0 


85.0 


66.7 


bl 


37.5 


60.0 


31.3 


21.1 


52.6 


20.0 


cl 


41.7 


66.7 


19.0 


35.0 


70.0 


23.1 


dl 


45.0 


62.5 


26.7 


45.0 


50.0 


0.0 


el 


59.0 


79.5 


62.5 


65.0 


85.0 


28.6 


fl 


66.7 


96.7 


100.0 


89.5 


100.0 


NA 


gl 


78.8 


93.9 


0.0 


83.3 


. 100.0 


NA 


hi 


47.4 


81.6 


57.1 


78.9 


89.5 


50.0 


al . 


50.0 


68.4 


47.4 


45.0 


65.0 * 


45.5 


bl 


34.2 


52.6 


32.0 


31.6 


42.1 


46.2 


cl 


41.7 


69.4 


33.3 


30.0 


75.0 


35.7 


dl 


70.0 


75.0 


70.0 


70.0 


70.0 


83.3 


el 


67.6 


81.1 


0.0 


85.0 


90.0 


100.0 


fl 


75.9 


89.7 


100.0 


81.3 


93.8 


100.0 


gl 


77.4 


96.8 


0.0 


82.4 


100.0 


NA 


hi 


48.5 


78.8 


71.4 


57.9 


84.2 


100.0 
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Appendix J 



Comparing Case Study Teacher Responses 
to First and Second Questionnaires 

(n=20) 



» 



» 



» 



» 



I 



ft 



Agreement on concept 



Agreement on duration 





Direct 

agreement 


Within one 
response 
category 


Direct 

agreement 


Within one 
response 
category 


item 

1 


(a) 


(b) 


(d) 


(e) 


2 










3 










4 a 


40.0 


70.0 






b 


10.0 


25.0 






c 


15.0 


30.0 






d 


35.0 


50.0 






e 


45.0' 


55.0 






5 


25.0 


55.0 






6 a 


55.0 


70.0 






b 


55.0 


100.0 






c 


50.0 


100.0 






7 a 


63.2 


89.5 


50.0 


85.0 


b 


42.1 


100.0 


55.0 


100.0 


c 


31.6 


89.5 


45.0 


95.0 


d 


52.6 


89.5 


70.0 


100.0 


e 


42.1 


89.5 


80.0 


100.0 


f 


31.6 


84.2 


65.0 


100.0 


g 


52.6 


78.9 


50.0 


95.0 


8 










9 a 


55.0 


80.0 


57.9 


89.5 


b 


45.0 


95.0 


45.0 


95.0 


c 


44.4 


88.9 


44.4 


100.0 


d 


47.4 


78.9 


52.6 


100.0 


e 


40.0 


75.0 


35.7 


100.0 


f 


45.0 


95.0 


55.0 


95.0 


g 


45.0 


80.0 


47.4 


94.7 


h 


47.4 


89.5 


52.6 


100.0 


10 a 


35.0 


85.0 






b 


75.0 


100.0 






c 


57.9 


84.2 






d 


22.2 


88.9 






e 


47.4 


94.7 






f 


50.0 


90.0 






g 


38.9 


88.9 






h 


26.3 


84.2 






i 


31.6 


78.9 






11 a 


95.0 


100.0 






b 


60.0 


95.0 






c 


40.0 


95.0 
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Comparing Case Study Teacher Responses 
to First and Second Questionnaires 

(n=20) 



Agreement on concept 



Agreement on duration 





Direct 

agreement 


Within one 
response 
category 


Direct 

agreement 


Within one 
response 
category 


tem 


(a) 


(b) 


(d) 


(e) 


d 


85.0 


95.0 






e 


55.0 


90.0 






f 


36.8 


73.7 






g 


31.6 


78.9 






h 


47.4 


94.7 






i 


50.0 


90.0 






j 


40.0 


85.0 






k 


45.0 


95.0 






1 


36.8 


84.2 






12 a 


47.4 


84.2 






b 


47.4 


94.7 






c 


70.0 


100.0 






13 a 


89.5 


94.7 


68.4 


94.7 


b 


60.0 


90.0 


68.4 


100.0 


c 


57.9 


89.5 


46.7 


100.0 


d 


25.0 


70.0 


80.0 


93.3 


e 


36.8 


84.2 


66.7 


100.0 


f 


60.0 


90.0 


63.2 


100.0 


g 


36.8 


84.2 


55.6 


100.0 


h 


42.1 


68.4 


61.5 


92.3 


i 


20.0 


100.0 


100.0 


100.0 


j 


30.0 


75.0 


31.6 


94.7 


k 


57.9 


100.0 


37.5 


100.0 


1 


50.0 


100.0 


46.2 


100.0 


14 a 


55.0 


100.0 






b 


45.0 


95.0 






c 


35.0 


90.0 






d 


60.0 


100.0 






e 


65.0 


95.0 






f 


85.0 


100.0 






g 


60.0 


90.0 






h 


45.0 


85.0 






i 


65.0 


100.0 






j 


50.0 


85.0 






k 


52.6 


100.0 






1 


42.1 


89.5 






m 


60.0 


95.0 






n 


65.0 


95.0 







o 

P 
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Appendix J 



Comparing Case Study Teacher Responses 
to First and Second Questionnaires 

(n=20) 



i 



» 



» 



i 



» 



tt 



Agreement on concept 



Agreement on duration 





Direct 

agreement 


Within one 
response 
category 


Direct 

agreement 


Within one 
response 
category 


item 


(a) 


(b) 


(d) 


(e) 


q 










15 a 


95.0 


100.0 


78.9 


94.7 


b 


73.7 


84.2 


77.8 


100.0 


c 


55.0 


75.0 


81.8 


100.0 


d 


45.0 


90.0 


52.6 


100.0 


e 


63.2 


84.2 


64.7 


94.1 


f 


42.1 


84.2 


70.6 


100.0 


g 


21.1 


84.2 


62.5 


100.0 


h 


78.9 


94.7 


78.9 


100.0 


i 


57.9 


94.7 


77.8 


88.9 


j 


68.4 


100.0 


66.7 


100.0 


k 


36.8 


73.7 


64.7 


100.0 


1 


78.9 


94.7 


83.3 


100.0 


m 


55.0 


90.0 


58.8 


100.0 


n 


66.7 


100.0 


50.0 


100.0 


0 


29.4 


76.5 


57.1 


100.0 


P 


27.8 


66.7 


42.9 


92.9 


q 


68.4 


94.7 


55.6 


100.0 


r 


55.0 


90.0 


63.2 


94.7 


s 


30.0 


75.0 


55.6 


94.4 


t 


85.0 


95.0 


61.1 


100.0 


u 


38.9 


83.3 


64.3 


100.0 


V 


50.0 


95.0 


58.3 


100.0 


w 


70.0 


100.0 


95.0 


95.0 


X 

16 


41.2 


58.8 






17 a 


58.8 


94.1 






b 


70.6 


100.0 






c 


62.5 


100.0 






d 


50.0 


93.8 






e 


58.8 


100.0 






f 


58.8 


100.0 






g 


64.7 


94.1 






h 


47.1 


76.5 






i 


35.3 


70.6 






j 


43.8 


93.8 






18 a 


95.0 


100.0 


73.7 


84.2 


b 


85.0 


100.0 


31.6 


89.5 


c 


85.0 


100.0 


52.6 


94.7 


d 


80.0 


100.0 


60.0 


75.0 
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Appendix J 



Comparing Case Study Teacher Responses 
to First and Second Questionnaires 

(n=20) 



Agreement on concept 



Agreement on duration 



item 



19 





Direct 

agreement 


Within one 
response 
category 


Direct 

agreement 


Within one 
response 
category 




(a) 


(b) 


(d) 


(e) 


e 


90.0 


100.0 


70.0 


95.0 


f 


68.4 


94.7 


57.1 


100.0 


g 


85.0 


100.0 


73.3 


100.0 


h 


73.7 


100.0 


47.4 


100.0 


a 


85.0 


100.0 


52.6 


89.5 


b 


89.5 


100.0 


44.4 


94.4 


c 


85.0 


100.0 


52.6 


94.7 


d 


80.0 


100.0 


75.0 


100.0 


e 


70.0 


95.0 


45.0 


80.0 


f 


47.1 


88.2 


69.2 


100.0 


g 


68.4 


100.0 


61.5 


84.6 


h 


83.3 


100.0 


52.9 


100.0 
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Description of Data Analyses 



136 



Classroom Instructional Practices: 

Measuring the Consistency and Reliability of Questionnaire Data 

Data Collected and Analyses Performed 



The project was designed to provide information that could be used to assess the validity and 
reliability of survey items on the Classroom Instructional Practices (CEP) Survey. To assess the items, we 
used information from the following instruments: 

Data Collected 



Questionnaire mailed to 418 eighth to twelfth grade mathematics teachers and completed by 296 

Classroom observation forms for 4 1 designated classes completed by observers and case study 
teachers 

Classroom logs of daily classroom instructional activities completed by 41 case study teachers for 
700 class sessions 

CEP survey completed by 41 case study teachers at the beginning of the case study 
CIP survey completed by 20 case study teachers at the conclusion of the case study 



Analyses Performed 



Using data from the above instruments, we conducted the following analyses to assess the validity 
and reliability of the survey items: 

Mail questionnaire item response rate 

Mail questionnaire item use rate 

Case study teacher and observer use of classroom log items 

Agreement between case study teachers and observers on concepts of student learning objectives 
and instructional activities 

Percent of teacher/observer nonagreement in which the teacher indicates the objective or activity 
did occur (and the observer indicates it did not) 

Agreement between case study teachers and observers on length of time for student learning 
objectives and instructional activities 



Percent of teacher/observer nonagreement in which the teacher indicates the objective or activity 
occurred for a longer length of time (than the observer indicates) 

Case study teacher use of classroom log items 

Consistency between case study teacher questionnaire responses and logs 

Percent of teacher survey and teacher log nonagreement in which teachers underreport on the 
survey 

Consistency between case study teachers’ first and second questionnaire responses 
The calculation and importance of each analysis is explained below. 
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Mail Survey Data and Analysis 



Mail Survey Item Response Rate (Appendix D, Column b) 

The item response rate is a measure of the proportion of all responses that is valid for each survey 
question. Examples of invalid responses include missing and inappropriate entries such as: (1) multiple 
responses for a single query; (2) responses that total to more or less than 100 percent (specifically for items 
4 and 8); and (3) non-numeric responses where numbers are called for (specifically for item 3). 

For each item, the response rate was calculated as follows: 



In two-part questions (items 9, 13, 15, 18, 19), respondents were instructed to answer part two 
only if their part-one response warranted (i.e., they used an instructional technique, an instructional 
material was available). For these items, inappropriate responses for part two include (in addition to those 
listed above) responses to part two where none was appropriate. Response rates for part 2 items were 
calculated as follows: 



The item response rate shows the level of item clarity, precision, and respondent compliance to 
survey directions. A low item response rate may indicate that: (1) respondents did not understand the item 
concept or wording; (2) respondents did not have sufficient knowledge or information to answer the 
question; or (3) respondents refused to answer the question. 

Mail Survey Item Use Rate (Appendix D, Column c) 

This calculation is a measure of the number of times that survey respondents indicated they used 
each instructional objective, method, or material, expressed as a percent of the total number of responses. 

The level of reported usage was calculated as follows: 



This measure identifies the most frequently-occurring classroom conditions, content, instruction, 
and materials. Items with a high percentage indicate that teachers use those most often; those with a low 
percentage are used least often. Certain objectives, activities, or materials may be incorporated into lessons 
more frequently because teachers may: (1) feel that certain methods lead to higher levels of success than 
others; (2) tend to use frequently methods or materials with which they are most comfortable; (3) be 
unfamiliar with alternative methods; or (4) not have access to alternative materials. 



total responses - (missing + inappropriate responses) / total responses 



total responses. - [Part 2 missing and Part 2 inappropriate responses) 
total responses - (Part 1 missing + Part 1 never) 



Total number item responses - number of “never” responses 
Total number of item responses 
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Case Study Data and Analysis 



Observation Data 

Case Study Teacher and Observer Use of Classroom Log Items ( Appendix G, Columns a, b, c) 

This calculation is a measure of the number of times during observations that teachers and/or 
observers indicated the teacher used each instructional objective, instructional method, student activity, or 
instructional materials (questionnaire items 10, 13, 15, 18, 19), expressed as a percent of the total number 
of observation logs. 

The level of reported usage was calculated as follows: 

Total number, oftimes item was checked by teachers and observers on observation logs 

Total number of logs completed 

This measure identifies the most frequently-used classroom instructional concepts and activities. 
Items with a high percentage indicate concepts, activities, or materials that teachers use most often; those 
with a low percentage are used least often. Certain objectives, activities, or materials may be incorporated 
into lessons more frequently because teachers may: (1) feel that certain methods lead to higher levels of 
success than others; (2) tend to use frequently methods or materials with which they are most comfortable; 
(3) be unfamiliar with alternative methods; or (4) not have access to alternative materials. 



Case Study Observer Use of Classroom Log Items During Observation ( Appendix G, Column b) 

Similar to the measure described above, this calculation is a measure of the number of times 
during observations that the observers indicated the teacher used each instructional objective, instructional 
method, student activity, or instructional materials (questionnaire items 10, 13, 15, 18, 19), expressed as a 
percent of the total number of observation logs. 

The level of reported usage by observers only was calculated as follows: 

Total number of times item was checked bv observers on observation loss 
Total number of logs completed 



Case Study Teacher Use of Classroom Log Items During Observation (Appendix G, Column c) 

Similar to both measures above, this calculation is a measure of the number of times during 
observations that the teachers indicated they used each instructional objective, instructional method, 
student activity, or instructional materials (questionnaire items 10, 13, 15, 18, 19), expressed as a percent 
of the total number of observation logs. 
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The level of reported usage by teachers only was calculated as follows: 



Total number of times item was checked by teachers on observation logs 
Total number of logs completed 



Case Study Teacher Use of Classroom Log Items during the Case Study Period (Appendix G, 

Column 2d) 

This calculation is a measure of the number of times during the four-week case study period that 
teachers indicated they used each instructional objective, method, activity, or material (questionnaire items 
10, 13, 15, 18, 19), expressed as a percent of the total number of classes for which we have log data. 

The level of reported usage was calculated as follows: 

Total number of times item checked bv^all case study teachers while completing four weeks of logs 

Total number of logs completed 

This measure identifies the most frequently-used classroom instructional concepts and activities. 
Items with a high percentage indicate activities that teachers use most often; those with a low percentage 
are used least often. Certain objectives and activities may be incorporated into lessons more frequently 
because teachers may: (1) feel that certain methods lead to higher levels of success than others; (2) tend to 
use frequently methods with which they are most comfortable; or (3) be unfamiliar with alternative 
methods. 



Agreement Between Case Study Teachers and Observers on Concepts of Student Learning 

Objectives and Instructional Activities ( Appendix H, Column a) 

This calculation measures the level of agreement on the understanding and recognition of concepts 
between case study teachers and classroom observers. 

The level of agreement was calculated as follows: 

number of times teachers and observers agree 
that a learning objective or instructional activity did or did not occur 
total number of logs 

This item reveals the extent to which teachers and observers demonstrated a similar understanding 
of the concept in question. Low levels of agreement may indicate that teachers and observers attach 
different conceptual meanings to the survey items or that teachers are unfamiliar with the item wording or 
concept. A high level of agreement may indicate that teachers and observers attach similar meanings to the 
concepts. 



m 



Percent of Teacher/Observer Nonagreement in which the Teacher Indicates Objective or 

Activity Did Occur (Appendix H, Column b) 

This item explores teacher/observer nonagreement on the occurrence of student learning objectives 
and classroom instructional activities. In those instances in which there is nonagreement, this item 
measures the percent of times in which the teacher indicates an objective or activity did occur and the 
observer indicates it did not occur. 

The level of teacher positive response was calculated as follows: 

where nonagreement, 

the number, of times case study teachers report occurrence 
total number of nonagreements between case study teacher and observer 

This item helps to illuminate the direction of nonagreement between case study teachers and 
observers for each objective or instructional activity. A high percentage of teacher positive response 
indicates that teachers consider the objective or activity to have occurred more frequently than observers. 

A low percentage indicates that observers think an objective or activity occurred more frequently than 
teachers. 



Agreement Between Case Study Teachers and Observers on Length of Time for Student 

Learning Objects and Instructional Activities ( Appendix H, Columns c, d) 

This calculation measures the level with which teachers and observers agreed on the length of time 
that student learning objectives and activities occurred in the observed class, after previously agreeing on 
the concept. It is measured at two levels: (1) the percent of direct agreement in which teachers and 
observers agree on the length of time it happened; and (2) the percent of time in which teachers and 
observers agree within one response category. 

The level of direct agreement (column d) was calculated as follows: 
number of exact agreements / number of logs 

The level of agreement within one response category (column e) was calculated as follows: 
number of exact agreements + number of agreements within one response category / number of logs 

These numbers show the level of overall agreement between teachers and observers on the length 
of the student learning objectives and instructional activities. High percentages suggest that teachers and 
observers had similar estimated times. Low percentages suggest that teachers and observers estimated time 
differently. 



Percent of Teacher/Observer Nonagreement in which the Teacher Indicates Objective or 

Activity Occurred fora Longer Period of Time (than the Observer Indicated) ( Appendix H, 

Column e) 

This item explores teacher/observer nonagreement on the length of time that the student learning 
objectives and classroom instructional activities occurred. In those instances in which there is 
nonagreement, this item measures the percent of times in which the teacher indicates an objective or 
activity occurred for a longer period of time than the observer indicates. 

The level of teachers’ report of longer time was calculated as follows: 

where nonagreement, 

instances in which teachers report longer time 
total number of nonagreements between case study teacher and observer 

This item helps to illuminate the direction of nonagreement between case study teachers and 
observers for each objective or instructional activity. A high percentage indicates that teachers think an 
objective or activity occurred for a longer period of time than did observers. A low percentage indicates 
that observers think an objective or activity occurred for a longer period of time than did teachers. 



Comparing Case Study Teacher Survey Responses with Log Entries 



Consistency Between Case Study Teacher Questionnaire Responses and Logs ( Appendix I, 

Columns a, b, d, e) 

This calculation measures the consistency of teachers’ survey responses on the first and second 
questionnaires with their daily logs of classroom instructional objectives and activities maintained over a 
minimum of four weeks. We compared the survey responses of individual teachers with their daily 
recordings. Consistency is measured at two levels: (1) the percent of direct agreement in which teachers’ 
survey responses and log recordings agree on whether an objective or activity occurred (or not) and the 
length of time it happened; and (2) the percent of time in which both recordings agree that an objective or 
activity occurred (or not) and the length of time on both recordings is within one response category. 

The level of direct agreement (column h) was calculated as follows: 
number of exact agreements / number of logs 

The level of agreement within one response category (column i) was calculated as follows: 
number of exact agreements + number of agreements within one response category /number of logs 

These numbers are a measure of the extent to which teachers’ survey responses reflect the type and 
extent of classroom activities as recorded on daily logs. High percentages suggest that the survey item is a 
reliable long-term indicator of daily class activities. Low percentages may indicate that teachers have 
difficulty accurately recounting overall class activities on surveys covering longer periods of time. 



Percent of Teacher Survey and Teacher Log Nonagreement in which Teachers Underreport on 

the Survey (Appendix I, Columns c, f) 

This calculation further explores teacher survey and log nonagreement by measuring the percent of 
time in which the teachers’ first and second survey responses indicate that an objective or activity occurs 
less frequently or for less time than their daily classroom logs indicate. 

The level of teacher positive response was calculated as follows: 

where nonagreement, 

the number of times daily log reports show more frequent or_ longer use than survey responses 
total number of nonagreements between survey responses and log reports 

This item helps to illuminate the direction of nonagreement between survey responses and daily 
activity logs A high percentage indicates that teachers’ survey responses underestimate the frequency or 
length of classroom learning objective or activity compared to what occurs on a daily basis. A low 
percentage indicates that teachers’ survey responses overestimate the frequency or length of classroom 
learning objective or activity compared to what occurs on a daily basis. 



Consistency Between Case Study Teachers’ First and Second Questionnaire Responses 

( Appendix J) 

This calculation measures the consistency of teachers’ first survey responses with their responses 
to the second survey. Consistency is measured at two levels: (1) the percent of direct agreement in which 
teachers’ two survey responses agree; and (2) the percent of time in which responses agree within one 
response category. 

The level of direct agreement was calculated as follows: 

number of exact agreements / number of surveys 

The level of agreement within one response category was calculated as follows: 
number of exact agreements + number of agreements within one response category / number of surveys 

These numbers show the level of overall agreement between teachers’ first and second survey 
responses. High percentages suggest that responses changed little over time; low percentages may indicate 
that keeping daily logs heightened teachers’ awareness of the frequency and duration with which 
classroom activities occur. 
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List of Items Deleted from Case Study Analyses 
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Appendix L 

List of Items Deleted From Case Study Analysis 



Following the preliminary analysis, we removed from analyses comparing case study teacher and 
observer responses certain sub-items for which little or no use by observers or case study teachers resulted 
in too few cases to assess. We eliminated those items receiving ten or fewer indications of use in the (82) 
records of the 41 classroom observations. This eliminated two of nine student learning objectives, four of 
12 teacher actions, six of 24 student activities, and four each of eight teacher and student instructional 
materials. 

The eliminated items are: 

Student Learning Objectives 

10c Collect data (e.g., observe, measure, count) 
lOd Order, compare, estimate, approximate 



Teacher Instructional Actions 

13c Demonstrate a concept, using three-dimensional tools such as manipulatives, models, or 
other objects 

13g Provide remedial or enriching instruction to a pull-out group while the rest of the class 
works on assignments 
1 3i Administer a test or quiz 
1 3k Demonstrate uses of technology in mathematics 



Student Instructional Activities 

15i Work on projects/assignments that take a week or more to finish 
15j Give or listen to other students give oral reports 
1 5 1 Write a report or paper 
15n Do lab or field work 

15v Participate in structured cooperative learning activities 
15w Take tests, quizzes, other assessments 



Teacher Instructional materials 
1 8b Graph paper 

1 8c Protractors, rulers, or compasses 

1 8f Film or videotape 

1 8g Computer or computer programs 



Student Instructional Materials 

19e Overhead projector 

1 9f Film or videotape 

1 9g Computer or computer programs 

1 9h Manipulatives, models, or other objects 
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Appendix M 



Summary of Item Recommendations 



Appendix M 



Summary of Recommended Changes to Fieldtested Items 



1 


Course title 


Keep, unchanged 


2 


Class schedule 


Keep, unchanged 


3 


Student grade levels 


Keep, unchanged depending on targeted respondents 


4 


Student academic abilities 


Revise to be close-ended that reflects the range of expected 
responses 


5 


Class capability of learning course 
material 


Revise to be close-ended that reflects the range of expected 
responses 


6 


Teacher influence on policy 


Delete, since similar to SASS item 44 


7 


Teacher control 


Limit to within classroom control 


8 


Course content 


Revise substantially to reduce burden and provide more depth, or 
delete 


9 


Skills and concepts taught 




10 


Student learning objectives 


Revise language to improve observer validation: 


11 


Assessment content 




12 


Interdisciplinary teaching 


Eliminate, unless specialized request for information 


13 


Teacher instructional activities 


Reduce number of subitems, revise wording of some subitems 


14 


Contextual factors 


Reduce number of subitems 


15 


Student learning activities 


Shorten stem; reduce number of subitems, revise wording of some 
subitems 


16 


Non-academic time 


Revise to be close-ended that reflects the range of expected 
responses 


17 


Demonstrate math competencies 




18 


Teacher materials 


Reduce number of subitems, combine with student materials 


19 


Student materials 


Reduce number of subitems, combine with teacher materials 
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